Quantifying Hope on a Global Scale

Hope is a some­what neb­u­lous word.

For some, it is an ex­pec­ta­tion of what the fu­ture will be. For oth­ers, it is a goal for what the fu­ture should be. The word it­self means some­thing dif­fer­ent to each per­son.

Most gen­er­ally though, hope de­scribes how one per­ceives the mo­men­tum of the world. Do things seem to be gen­er­ally get­ting bet­ter, or do they gen­er­ally seem to be get­ting worse? Per­son­ally, I think things are rapidly get­ting much, much bet­ter.

We Should Be Hopeful

First, let’s de­fine our met­ric. The world is bet­ter off when peo­ple are bet­ter off. The more peo­ple who are able to live long, mean­ing­ful lives, the bet­ter.

In his posthu­mously pub­lished book, Factfulness: Ten Reasons We’re Wrong About the World — and Why Things Are Better Than You Think, Hans Rosling made the case that the qual­ity and length of hu­man life has been trend­ing up­ward for more than a cen­tury. Life ex­pectancy at birth has more than dou­bled in the last cen­tury alone. The por­tion of the global pop­u­la­tion liv­ing in poverty is lower than any point in his­tory.

Side note: a great re­source to learn about this is Our World In Data’s com­pre­hen­sive, yet ap­proach­able re­port.

All of this progress does not mean we should just stop here and say the work is “done.” However, I be­lieve that of­ten the best ap­proaches to solv­ing world-scale prob­lems (like poverty) start by first look­ing at the so­lu­tions that al­ready work.

In the book, Rosling tells how he is of­ten de­scribed as an “optimist.” He thinks it is a mis­nomer, since op­ti­mists are of­ten por­trayed as fool­ish and mis­in­formed. He, how­ever was likely more qual­i­fied than any­one to speak on the pos­si­bil­i­ties of the likely good fu­ture, so his pos­i­tive out­look was en­tirely jus­ti­fied. For this rea­son, he de­scribed him­self a “possibilist”. While I don’t even ap­proach his qual­i­fi­ca­tions, I would sim­i­larly de­scribe my world­view as “possibilist.”

The Solutions That Exist

I’ve spo­ken on the topic of so­lu­tion-fo­cused think­ing be­fore, but I think it war­rants re­vis­it­ing. I don’t want this to be the fo­cus of my writ­ing to­day, but I do want to pro­vide a brief ex­am­ple of the kind of so­lu­tions that ex­ist to­day that are hav­ing pro­found, real-world im­pacts.

One of the crit­i­cal en­vi­ron­men­tal prob­lems is re­lated to re­cy­cling. Every year, 18 mil­lion tonnes of trash is leaked to the en­vi­ron­ment, in­clud­ing our oceans and rivers. Ac­cord­ing to AMP Robotics founder Matanya Horowitz, this trash can ac­tu­ally be a hugely prof­itable com­mod­ity. De­pend­ing on mar­ket con­di­tions, one ton of re­cy­clable ma­te­r­ial can be worth up to $100. The is­sue: the cost of sort­ing the ma­te­r­ial into a us­able form is of­ten also around $100, most of which is spent on hu­man la­bor. Since the cost of pro­cess­ing the ma­te­r­ial to pre­pare to sell is about the same as it’s sold price, there is lit­tle eco­nomic in­cen­tive to do much re­cy­cling. In the case of plas­tics re­cy­cling, the is­sue is not that peo­ple don’t re­cy­cle enough. It is that the cost of re­cy­cling the re­sult­ing ma­te­r­ial is too high.

So what does AMP Robotics do? They have au­to­mated the process of re­cy­cling, bring­ing down the costs pre­cip­i­tously. With ro­bot­ics, the cost of re­cy­cling (to the waste com­pany) is now much lower than the ac­tual sold price of ma­te­r­ial, mak­ing it an ex­tremely prof­itable busi­ness.

I think this is a great ex­am­ple for two rea­sons. First and fore­most, we are of­ten told, with­out much sub­stan­ti­at­ing ev­i­dence, that the prob­lem of ocean plas­tics and pol­lu­tion are the re­sult of in­di­vid­ual peo­ple choos­ing not to re­cy­cle. In re­al­ity, how­ever, the lim­it­ing fac­tor is not on the in­di­vid­ual, but the waste man­age­ment com­pa­ny’s ef­fi­ciency.

Quantifying Hope

I’ve es­tab­lished that the world is be­com­ing a bet­ter place to live for al­most every­body, all the time. I’ve also made clear that this pos­i­tive change is the re­sult of so­lu­tions that get re­ported on (by the news) less fre­quently than the prob­lems they solve. With that done, I think it is time to re­turn to the topic of hope.

My end goal was to au­to­mate the process of find­ing so­lu­tions that worked (like AMP Robotics). I un­der­stood that this would be an enor­mously dif­fi­cult nat­ural lan­guage pro­cess­ing task, so I de­cided to start sim­ple. The plan: I would first quan­tify how hope­ful a given news ar­ti­cle was, then work from there.

Web Scraping

The first step was to write a sys­tem that could re­trieve re­cent, im­por­tant news ar­ti­cles re­li­ably. There is al­ready in­fra­struc­ture on most news web­sites that makes this pretty easy: RSS. By reg­u­larly it­er­at­ing through a list of RSS feeds, we can get pretty good news cov­er­age for most of the world. Ad­di­tion­ally, news or­ga­ni­za­tions pri­or­i­tize the con­tent of their RSS based on what they think is im­por­tant.

My ini­tial im­ple­men­ta­tion was in Rust, which I will show here. As you can see, I spent some time par­al­leliz­ing the scrape job. This was pri­mar­ily to re­duce the amount of time it would take to it­er­ate on the hope quan­ti­fy­ing al­go­rithm, so I could get close to real-time feed­back on it when I made changes.

use rss::{Channel, Item};
use tokio::sync::mpsc::unbounded_channel;

/// Download all items from an RSS Feed
async fn get_feed_url_items(url: &str) -> anyhow::Result<Vec<Item>> {
    let content = reqwest::get(url).await?.bytes().await?;

    let channel = Channel::read_from(&content[..])?;
    Ok(channel.into_items())
}

/// Download all items from a list of RSS feeds, in parallel.
pub async fn get_all_feed_urls(urls: impl IntoIterator<Item = &str>) -> anyhow::Result<Vec<Item>> {
    let mut items_receiver = {
        let (items_sender, items_receiver) = unbounded_channel();

        for url in urls {
            let url = url.to_string();
            let items_sender = items_sender.clone();
            tokio::spawn(async move {
                items_sender.send(get_feed_url_items(&url).await).unwrap();
            });
        }
        items_receiver
    };

    let mut res = Vec::new();

    while let Some(items_res) = items_receiver.recv().await {
        let mut items = items_res?;
        res.append(&mut items);
    }

    Ok(res)
}

Rewriting into .NET Core

Now that I’ve found a rel­a­tively high qual­ity source of raw data to es­ti­mate pub­lic opin­ion, I need to con­dense it into a sin­gle num­ber, and look at how it changes over time. It was at this point I rewrote most of the code into C# and ASP.NET Core. I had four rea­sons for this:

1. While I’ve used ASP.NET Core and Entity Framework Core in the past, I haven’t used it re­cently. I wanted to see how the ecosys­tem has changed in the in­ter­ven­ing time.
2. Evaluate the tech stack for other pro­jects.
3. Entity Framework Core is a great ORM, and I may be able to ap­ply its ide­olo­gies to other tech­nolo­gies, even if I don’t con­tinue us­ing it di­rectly.
4. In or­der to eval­u­ate pos­i­tiv­ity over time, I need as much his­tor­i­cal data as pos­si­ble, which should be stored in a per­sis­tent data­base.

One main draw­back of ASP.NET Core is the amount of boil­er­plate nec­es­sary. As a re­sult, I am hes­i­tant to show any clip­pings from the rewrite, as it takes a lot of space with­out adding much value to the dis­cus­sion here. How­ever, all the code for this pro­ject is open source and avail­able on GitHub.

Here is the gist of the over­all process the re-writ­ten app goes through:

1. Scrape a num­ber of RSS feeds with the method al­ready dis­cussed.
2. Store head­lines in a table in an SQLite data­base, along with time­stamp of the re­trieval and the source URL.
3. Repeat the first two steps every two min­utes, check­ing and re­mov­ing du­pli­cates.

Finally: The Hope Algorithm

Finally, we can dis­cuss the ac­tual al­go­rithm.

First, I would run the VADER sen­ti­ment analy­sis al­go­rithm over each head­line from the last month to eval­u­ate their pos­i­tiv­ity. Since VADER re­turns three val­ues, a “positivity” rat­ing, a “negativity” rat­ing, and a “neutral” rat­ing, I con­sol­i­dated them into a sin­gle num­ber by sim­ply sub­tract­ing neg­a­tiv­ity from pos­i­tiv­ity.

$$\text{headline rat­ing} = \text{positivity} - \text{negativity}$$

Next, I find the av­er­age of the sub­set of the pos­i­tiv­ity score from the past two hours and sub­tract the av­er­age from the last month. Fi­nally, I use the mon­th’s stan­dard de­vi­a­tion to place it on a scale of 0 – 100, where 50 is av­er­age. Here is the for­mula, where $\bar{x}_h$ is the pos­i­tiv­ity mean from the last two hours and $\bar{x}_m$ is the pos­i­tiv­ity mean from the last month.

$$\text{hope score} = 200\left(\frac{\bar{x}_h - \bar{x}_m}{\sigma_m} \right) + 50$$

Naturally, there are a few things to note about this for­mula. I as­sumed that the pos­i­tiv­ity score is a nor­mal dis­tri­b­u­tion among news head­lines (I did not have time to eval­u­ate whether this is ac­tu­ally true). The score will also of­ten fall out­side the range of 0 – 100. This is ex­pected, since val­ues out­side that range will be less ac­cu­rate any­way. In those sit­u­a­tions, I sim­ply clamp to within the range 0 – 100.

Communicating Hope

I wanted to ex­plore and get feed­back on a cou­ple dif­fer­ent ways of com­mu­ni­cat­ing ab­stract con­cepts of hope. First, I threw to­gether a quick web­site us­ing Svelte and TailwindCSS.

I ran the web­site for a cou­ple of days and had friends, fam­ily, and peers test it out to get feed­back on the con­cept.

Feedback

The peo­ple I spoke to had a lot to say on the dial it­self. No­tably, the lack of +/- sig­nal­ing was con­fus­ing. Peo­ple could­n’t tell what a “good” or a “bad” read­ing on the dial looked like. One per­son sug­gested I in­clude a happy face on the hope­ful side and a sad face on the de­spair side.

We brain­stormed the dif­fer­ent places we could put a hope-me­ter. IOS and Android both sup­port home-screen wid­gets. Maybe we could put a global hope-me­ter on one of those, right next to the clock? Maybe you could click on a spe­cific spot on the hope me­ter, and you would be served a news ar­ti­cle with that amount of hope? The pos­si­bil­i­ties were end­less. I was pushed to look into what a phys­i­cal hope me­ter would look like, so I sketched it out.

Moving To the Real World

I got hooked on the phys­i­cal dial. I wanted some­thing tan­gi­ble that I could put some­where. Some­thing whose phys­i­cal weight would give an amount of sub­stance to the thing it mea­sured.

I de­cided to keep riff­ing on the sketches I had drawn out. I loaded the web server I used for the orig­i­nal site onto a Raspberry Pi 3, plan­ning on us­ing the ex­ist­ing HTTP end­point to con­trol a servo, which in turn, moved a hand on a dial.

The only ma­te­r­ial I needed to buy was a FS5103R con­tin­u­ous ro­ta­tion servo. I al­ready had ac­cess to every­thing else. While I was wait­ing on that part to ar­rive, I mod­eled out what the phys­i­cal de­vice could look like. Ini­tially, I imag­ined some­thing like this:

All the elec­tron­ics would be placed in­side the bot­tle, and the servo would be at­tached to the cap. The en­tire con­trap­tion would rest on it’s side. In hind­sight, there are a cou­ple of is­sues with this con­cept. The cir­cu­lar body would cause it to roll all over the place, and the com­plete lack of ac­cess to the in­side would make main­te­nance quite dif­fi­cult. That is­n’t even to men­tion there is­n’t a sin­gle hole for a power cord in this model for the elec­tron­ics.

In the fi­nal model (pictured above), I in­cluded fa­cial ex­pres­sions to dif­fer­en­ti­ate hope and de­spair on the dial. I also com­pletely re­moved the en­clo­sure, in­stead plan­ning on di­rectly at­tach­ing the Raspberry Pi and servo to the back of the dial.

The Print

With the model com­plete, I went to my lo­cal mak­er­space, and printed it out. How did it go? Not so well.

The first print failed spec­tac­u­larly. Parts were com­ing off, and it looked like it was drawn with one of those 3D pens. Turns out, the is­sue was that the un­der­ly­ing 3D model had not been ex­ported out of Blender (the 3D mod­el­ing soft­ware I used) prop­erly, and the geom­e­try was screwed up be­fore it even en­tered the slic­ing soft­ware. Garbage in, garbage out, I sup­pose.

The sec­ond print, how­ever, turned out great. I hot-glued the Raspberry Pi to the back, plugged in the servo, and we were off to the races. As you can see above, I pro­grammed a lit­tle servo con­trol script in Python, which in­cluded a lit­tle startup mo­tion. Af­ter the mo­tion is com­pleted, it moves to the cur­rent hope value.

The video was recorded right af­ter the Isreal-Hamas War be­gan, so global hope val­ues were not do­ing so hot. Al­though it truly is a tragedy, if any­thing, this is ev­i­dence that the sys­tem works.

Reflection

This pro­ject stretched my abil­i­ties in a cou­ple key ways.

After the ini­tial pro­to­typ­ing phase, all the code was in pro­gram­ming en­vi­ron­ments I am rel­a­tively un­fa­mil­iar with. While I have a lot of ex­pe­ri­ence with Node.js, Rust and React, I felt some dif­fi­culty when work­ing with ASP.NET Core and Python. The dif­fi­cul­ties were ex­ac­er­bated when try­ing to run every­thing on a Raspberry Pi, which I have never worked with in this ca­pac­ity.

Additionally, I found it pretty dif­fi­cult to get the servo work­ing re­li­ably. Ap­par­ently, ser­vos rely on a po­ten­tiome­ter to stay cal­i­brated. If the po­ten­tiome­ter moves to an am­bi­ent en­vi­ron­ment with a dif­fer­ent tem­per­a­ture than it was cal­i­brated in, the servo starts to move in un­ex­pected ways. This re­sulted in fre­quently in­ac­cu­rate read­ings. If I could go back and re-do this pro­ject, I would have used a dif­fer­ent kind of mo­tor, maybe a step­per mo­tor.

Part of my mo­ti­va­tion for do­ing this pro­ject was to gather more in­for­ma­tion on the best way to im­ple­ment a cli­mate change progress bar. To that end, I found this quite suc­cess­ful. I now have a bet­ter un­der­stand­ing of what kind of server and data­base ar­chi­tec­ture I want to use. I also now know what kinds of for­mats peo­ple are more likely to see global met­rics like hope or cli­mate progress (home screen wid­gets and push no­ti­fi­ca­tions are at the top of the list).

Note to the Grandey Honors Program: I know you want a for­mal self as­sess­ment. This is it. I be­lieve I de­serve a “T”. My re­flec­tion and port­fo­lio are quite pol­ished and con­vey the story of build­ing my Hope-o-meter rel­a­tively well. I demon­stra­bly went out­side my com­fort zone and at­tempted to solve (what I per­ceive to be) a real world prob­lem. My work should stand on its own, so I feel no fur­ther need to jus­tify a “T”.