It’s been quite a while (just a few days more than a year!) since the first post about my project to build a system that identifies birds at a bird feeder using computer vision. The project has been progressing in bursts since then, and for the past 6 months I’ve had it in the back of my head to write an update post, so here it is. This post goes roughly in chronological order but I’m going off memory so it may not be entirely accurate.

Livestream

The feed has been streaming live on Twitch for the past year, have a look!

Setup

The first major change after the initial blog post was ditching the network-based video streaming setup. It was unreliable and required running lots of software on both the camera host as well as my desktop, and since it used my desktop GPU for inference it limited what I could use my desktop computer for - not to mention the stream going down every time I rebooted my computer.

After deciding that I wanted to maintain this as a long term project I purchased a NUC and an eGPU enclosure. I initially tried to use the enclosure with an RTX 3070, but I couldn’t get it working with that card so I used a 1070 that was lying around instead which worked flawlessly. The 1070 runs at about 25fps when inferencing with my bird model which is more than enough. The whole thing sits on my kitchen floor and is relatively unobtrusive.

What I was doing before was simply streaming the window displayed by YOLO’s detect.py convenience script. However, at 25fps this isn’t a great viewing experience. I wanted to stream 30 or 60fps video and overlay the labels on top of it. Doing this turned out to be rather difficult because you cannot multiplex camera devices on Windows; only one program can have a handle on the camera and its video feed to the exclusion of all others. Fortunately there is some software which works around this [0]. I purchased that software and used it to split the camera feed in two. OBS directly consumes one feed and the other one goes into YOLO for inferencing. The resulting labeled frames are displayed in a live preview window. I then patched YOLO so that the preview window, which normally displays the source frame annotated with the inferencing results, only displayed the annotations on a black background without the source frame. That window is used as a layer in OBS with a luma filter applied to make the black parts transparent. With some additional tweaks to get the canvas sizing and aspect ratio correct this allowed me to composite the 25fps inferencing results on top of the high quality 60fps video coming from the camera.

For encoding I use Nvenc on the 1070. That keeps the stream at a solid 60fps, which the NUC CPU can’t accomplish. Between inferencing and video encode the card is getting put to great use.

[0] This method no longer works on Windows 11

Camera

The original iteration used an off brand 720p webcam wrapped in a righteous amount of plastic wrap for weatherproofing. Surprisingly the weatherproofing worked well and there was never a major failure while using the first camera. However, the quality and color on that camera wasn’t good and an upgrade was due. I already had a Logitech Brio 4k webcam intended for remote work, but it ended up largely unused so it was repurposed for birdwatching.

While the plastic wrap method never had any major failures it wasn’t ideal either. Heavy humidity created fogging inside the plastic that could take a few hours to wear off. It needed replacing anytime the camera was adjusted. Due to these problems and the higher cost of the Brio I decided to build a weatherproof enclosure.

The feeder is constructed of acrylic. My initial plan was to use acrylic sheeting build out an extension to the feeder big enough to house the camera. I picked up some acrylic sheeting from Amazon and began researching appropriate adhesives. It turns out most adhesives don’t work very well on acrylic, at least not for my use case – the load bearing joints between the sheets were thin and I needed the construction to be rigid enough to support its own weight and the weight of the camera without sagging. Since the enclosure would be suspended over air relying on its inherent rigidity for structure the adhesive needed to be strong.

The best way to adhere acrylic to itself is using acrylic cement. Acrylic cement dissolves the surfaces of the two pieces to be bonded, allowing them to mingle, and then evaporates away. This effectively fuses the two pieces together with a fairly strong bond (though not as strong as if the piece had been manufactured that way).

Three sides were opaque to prevent sunlight reflections within the box. Joints were caulked and taped the joints to increase weather resistance. I played around with using magnets to secure the enclosure to the main feeder body but didn’t come up with anything I liked, so I glued it to the feeder with more acrylic cement, threw my camera in there and called it a day.

This weatherproofing solution turned out great. It successfully protected the camera from all inclement weather until I retired that feeder, surviving rain, snow, and high winds over the course of the year.

Weather

I thought a weather widget might be cool, so I added it. Someone has nice HTML weather widget which is what I use.

Perch

I noticed there was quite a lot of competition for space on the rim of the feeder. I decided to add a perch thinking that the birds would appreciate more space, and I would get a better viewing angle if they chose to use it.

The perch was made out of a wooden dowel from the hardware store and secured to the feeder initially with duct tape. The duct tape worked until mourning doves started using the perch; they’re heavy enough that the perch would sag.

This was solved by mounting the dowel with a metal bracket screwed into the feeder body. I couldn’t find a bracket small enough to snugly mount the size dowel I got (or any size close to it), so to solve this problem I wrapped one end of the dowel in duct tape until it reached the requisite thickness.

This solved the overweight dove problem.

Attaching the dowel was fairly straightforward, but the screws had sharp points that ended up on the inside of the feeder body. I anticipated a startled bird catching on the screw tips and injuring itself. To prevent this I used my Dremel to grind down the bits.

The perch was a huge hit, the birds loved it. After installing the perch titmice and chickadees would grab a sunflower seed, jump to the perch, smash the seed open and eat the meat, all in view of the camera. Birds also started lining up on the perch and taking turns, making for an entertaining stream.

Metrics

I was curious how many birds were visiting per day, what peak hours were and what the species distribution looked like. In order to get this information the first step was data collection, so I thought about how to best collect data on visitors. Conceptually I wanted to log each visitor along with some relevant data such as species, time of visit, length of visit etc. as a single record – one per visitor. However, this is one of those problems that seems easy but is actually quite hard. The classification algorithm runs once per video frame but there is no context retained between frames; it can’t tell you that the bird detected in frame N_1 is the same one it detected a moment ago in frame N_0. Extracting this sort of context is probably feasible, perhaps by combining some heuristics with positional information (is the bird of species X in this frame in the same position as bird of species X in the previous frame? If so, probably the same bird). However, this fell in the realm of “too hard for this project.” Instead of storing bird visits I decided to store the data in its elementary format as a time series of detection events.

I have some familiarity with time series databases, having used InfluxDB for a different project. Influx was nice for that project but I wanted to learn something new so I chose Timescale this time around. Timescale is an extension to Postgres and I wanted to learn more about Postgres anyway so it was a good choice from a learning perspective. Since the plan was to insert a new record for every frame where an object was detected and the detection loop runs at about 25fps I was expecting a lot of data. I had read that Timescale handles that sort of scale a bit better than Influx, so that was another factor in choosing Timescale.

After setting up Timescale on a VPS (see how) it was time to design the database schema. I had a vague understanding of how relational databases worked and had written a few lines of SQL, but otherwise lacked database experience. After writing out a naive (in retrospect) schema it occurred to me that there was probably a way to do it right. I began learning about schema design and database normalization, a topic I found surprisingly interesting. This is the schema I ended up with, and while I don’t think it’s perfectly normalized it’s better than what I had initially.

Note the data for the feed present in the feeder. I thought it would be cool to correlate feed with frequency of species, but since I set things up so that changing the feed required a manual record insert I got lazy and stopped doing it so I don’t have the dataset needed now.

After setting up the database on the server, I made some hacks in the YOLOv5 code to push detection data directly to the database whenever a detection occurred. After a short time I had tons of visitor records in my database. Next I wanted to answer questions like “what is the most common species over the last week?” and “what time of day has peak volume?” To accomplish this I set up a Grafana dashboard.

Then I thought it would be cool to show these graphs on the livestream. It turns out Grafana supports embedding individual graphs, and since OBS supports rendering browser views it was easy to get those set up.

I left these up for a while, but ultimately I felt they were taking up too much space in the stream so I took them down.

New Feeder Era

After nearly full year of heavy duty service, one day I went to refill the feeder. This requires removing it from the suction cup mounts. While doing this, I broke the front of the feeder off.

By that point the feeder was already in pretty rough shape. Large parts of it had been sanded, drilled, and cemented, and it was scratched up from bird talons and stained with bird excrement. It did its duty with honor but it was time to replace it. I replaced it with a different design, still a simple acrylic feeder, but this time with a sloping roof. I wanted to experiment with extending the sloping roof to create a simple tent-style weatherproof shelter for the camera that didn’t require constructing a full box.

With the benefit of experience and the simpler design it only took a couple hours to create the appropriate acrylic slabs, cement them in place, caulk the joints and remount the camera.

This feeder has been in service for about 3 months and the new shelter is holding up much better than the previous one. Water runs right off it.

The new close up angle has been getting some great shots as well!

Additional thoughts

I am working on getting my hacky changes to YOLO and my data processing code whipped into shape for a rerelease; it’s currently all on my github but not easy to use. I’ll probably post on my social page when I do.

This has been one of the best projects I’ve ever worked on. I have learned to identify so many common birds, been exposed to so many new to me technologies and heard from lots of people who really enjoy the stream. It’s been rewarding in so many different ways.

Happy birding!