Zyra's front page //// Concepts //// How to find stuff //// How to find stuff on Zyra's website //// Site Index

Needle in a Haystack

If something is difficult to find, it may be described as "like trying to find a needle in a haystack". The analogy is particularly apt where a major reason for the difficulty is in the way the item being looked-for is hidden by the background being similar to the sought-item. A needle, being a small object visible as a short straight line, is very hard to see against a background such as that presented inside a haystack, which has many criss-crossing pieces of hay which also look like little straight lines.

The expression "like trying to finding a needle in a haystack" is quite old, and dates back to the times when people were familiar with agriculture and handicrafts at least to the level of knowing that milk comes from cows rather than being manufactured in factory, and that "a stitch in time saves nine" refers to the idea that clothes can be repaired.

Some examples of searches that may be appropriately described as like looking for a needle in a haystack:

* Trying to find a dropped contact lens, especially if it was lost on a large mottled floor surface.

* The search for small planets in the vast expanse of the starry sky.

* The hunt for a murder weapon in the murk of a deep dark lake, especially if miscellaneous junk had been dumped in the lake over the years.

* Trying to recover a tiny gemstone which you've vacuumed up, thus requiring a very careful picking-over the contents of the dust-bag of the vacuum cleaner.

The defining features of a needle/haystack type search task are:

1. That there's a big expanse of search area to sift through, in comparison to the relative small size of the object.

2. That the nature of the background is such that the object is to some extent camouflaged, resulting in false "found it!" alarms.

To get an approximation of how difficult a literal interpretation of the "needle in a haystack" search would be: If a haystack contained ten tonnes of the dry stalks off grasses known as "hay", and the needle in question was an average-sized sewing needle, then searching for the needle in the haystack would be 1000 times more difficult that finding the same needle in 10 kilogrammes of hay, ie a large bagful. If you want to do a practical experiment on this, get some hay and try it. (Hay is available from horsey people). If it takes you 10-20 minutes to find the needle in a big bag of hay, then it would be several weeks of painstaking work literally to find a needle in a haystack, if you did it "the hard way".

Of course, if you had to complete a "Herculean Task" of recovering a lost needle from inside a haystack, you'd most likely want to use a clever method, something which gets around the classic difficulty of the task. In practice, here are a few ideas:

* Use a huge magnet. If you used a scrap-merchant's giant electromagnet on a crane, of the type used for lifting scrap cars, and had the hay being pitchforked onto an agricultural quality conveyor belt, there would come a point when the needle leapt out of the hay and stuck to the magnet. Another variation would be to spread the entire haystack over a disused airfield, and then drive the crane to and fro with the magnet powered up.

* Use a metal detector. This sounds easy, but in practice the electromagnetic permeability effects associated with a needle are not great, so it would require some close proximity.

* Float the hay on water. As hay mostly floats on water, and metal needles sink, you could have the hay spread over a giant pool or millpond, and after a while the needle would sink. The downsides of this method include the fact that the hay would be wet and would make a terrible mess, and you'd still have to find the needle at the bottom of the water.

* Have the entire haystack loaded into a shipping-container and sent to an enterprising company in the Third World, where the task of finding the needle could be subcontracted and a large workforce employed cheaply. Even if you made sure everyone was paid more than the average wage for that geographical region, it would still be more economically viable than might be expected.

* Organise a competition and advertise it. Invite interested people by such tempting and interesting offers as "Search for a Needle in a Haystack! Cash Prize to whoever finds it!". You'll get more people taking up your offer than if you advertised "Wild Goose Chase to be held on [such and such a date] at [some place or other]"! Because of the gambling mentality of folk, and the iconic nature of "needle in a haystack" and the fact that it's almost never used in a literal sense, there would be volunteers, and the media coverage would be good!

* Use radio-waves. Hay is a dry material through which radio waves will pass, whereas a metal needle will resonate at a particular wavelength and will appear as a pinging trace like on a radar. If you happened to know the length of the needle it would help, as that would indicate the wavelength, and you'd know what frequencies to send by virtue of the relationship between radio frequency and wavelength

* Use a distributed an open-source method! In this approach, rather than one person putting a concentrated effort into the needle-haystack task, the hay is forked into thousands of bags and they are sent out to be dealt with by a great many people on a distributed network. The technique has already been used by SETI in their Search for Extraterrestrial Intelligence, by getting people around the world to load the SETI-at-home screensaver background task, thus applying more computer power to the task than could be obtained from the most powerful computers on Earth. The SETI data sifting program is a classic example of a "needle in a haystack" task.

* Burn the haystack. Obviously not recommended, because it would not be good environmentally, and it would also be dangerous, and wasteful. The amount of heat produced would be immense, as it would burn with an energy output of several megawatts for a while. If the needle was steel it would probably survive, but the tempering of the metal would not be so good, and you'd still have to find it amongst loads of ash.

* Use a wind tunnel. I'm sure Dyson would have some insight into this. At the right airflow rate, the lightweight hay would be blown away down the tube, whereas the higher density needle would be left to fall. This is a "separating the wheat from the chaff" solution, although I don't usually talk about wheat in that sense because I'm a coeliac

* Put the whole lot through the mill. Based on the idea that straws break but needles stick into things, a suitably alternative designed grinding mill with soft rollers would crush all the hay into a still-usable animal-feed and would end up with the needle impaling a pincushion-like component of the design.

* Feed the hay to small animals. They'd have to be small, because larger beasts would be injured when they accidentally ate the needle. Something more the size of insects or mice would avoid the needle. The practicality of the idea is put into question somewhat as creatures that eat also have an output, so the solution would have to address that problem too.

Hay is usually fed to animals, but with the "needle" problem, it's important to realise that some animals are very careful when they eat, and some are not. The ideal animal for separating a haystack from a needle would be an animal that is sensitive in its eating style and won't ingest the needle. How about chickens ?

* Rot the stack. Use compost. This would be OK only if the needle was entirely rustproof and you didn't mind it taking months. Hay is organic and will eventually turn to soil, whereas the metal needle will survive.

* Use nanotech. Micro-robotics would make light work of the needle/haystack problem, probably so well that salespeople might use it as a demonstration of the prowess of their product! Like a futuristic equivalent of late twentieth century washing powder commercials, the "before" situation would be shown, ...and then we just add a few mega of microbots... , and in a trice "needle found!" comes up on the network screen.

Other needle/haystack references:

http://en.wikipedia.org/wiki/Needle_in_a_haystack - Needle in a haystack given a mention at Wikipedia

Needle in a Haystack http://www.needlestack.com/ - Needlestack - purveyor of fine needlework supplies. I wonder if they'll have an affiliate program?!

Needle in a Haystack http://www.needhay.com/ - Needhay - fine yarn and needlework supplies - same question applies

Needle in a Haystack - http://www.needleinahaystack.biz/ - needlepoint - fine paper - especially Wedding invitations

http://idioms.thefreedictionary.com/needle+in+a+haystack - the idiom briefly mentioned at Free Dictionary site, in amongst a haystack's worth of vaguely irrelevant ads (gone)

http://perspectives.mvdirona.com/2008/06/30/FacebookNeedleInAHaystackEfficientStorageOfBillionsOfPhotos.aspx - MySQL and "effective storage of billions of photos"

...and other resources which will most likely be added when we can FIND them!

Other useful things:

How to find stuff on Zyra's website

How to find stuff - general guide

Further reading: If you're doing some research into the origins of the "needle" and "haystack" notion, you may be interested in the following. A helpful correspondent has written in and pointed out "With reference to looking for a needle in a haystack it is not such a laborious job as a haystack needle is in itself quite large as they used it when thatching the haystack to keep the weather out". That's helpful!

Now of course you can't search on search engines directly for "haystack needle" and get much success because it's like...(!)... well it's difficult to find. However, our correspondent has a good point about the thatching of haystacks, and it makes sense as thatched roofs on houses work at keeping out the wet, so the thing to search for is "thatching needle". This gives a variety of interesting results showing the type of equipment that a roof thatcher, and therefore a haystack thatcher, would use. The implements in question are large, akin to the hefty needles which fishermen use for mending trawl nets. Also consider what type of sewing tackle someone would use to knit a football goal net.

So, not a sewing needle for embroidery, but a thatching needle for putting a roof on a hay rick.

This is one of the best pages about "Needle in a Haystack" anywhere on the Internet. Some places get this right, and some don't. Well done to Yandex.ru for getting this right! (2012/09)