[Devember 2022] Mastocritter, US politician stock trades in Mastodon [COMPLETE]

So I did the stonkcritter entry last time, but it was kind of vapourware since I couldn’t keep it going due to some random bug I couldn’t find where it would randomly stop updating the file. Feel pretty stink about that especially after the guys were nice enough to post a motherboard to the bottom of the planet as prize.

I really liked the subscription feature though and I put a lot of work into that, which has helped me a lot with other Telegram bots. I also experimented with recreating the entire thing in NodeRed which was kind of OK - great platform, pain to productionize.

However, what about making the politicians stock trades more accessible, so you don’t need a Telegram account to access them? Especially if it could be ActivityPub’d across the internet

In the middle of the year I added a renderer to stonkcritter that would take the stock trades JSON file and dump out ruby code that could be loaded into a mastodon instance via the rails console. I even got this running locally in a mastodon instance and it was pretty cool, so I’ve already proven the concept. I actually thought I lost this code but was able to restore it from an NVME that I thought had died.

With the renewed interest in mastodon this should be useful to some people who would like to pull such info into their own instances. I also have some as yet untested avatar generators too thanks to the congress website.

So this project will be about getting the Mastodon instance and productionizing that updating process, as I was able to get the whole dump in, but how about updating the database daily?

Perhaps a side effect of this would allow me to use the same update method to feed into stonkcritter to get it working for more than a few days too.

As a stretch goal I’m also interested in generating different kinds of reports, like which critters report the fastest, and which the slowest etc. This could be added to a post as well, but as an image. Suggestions here would be welcomed.

Not sure how far I will get because I have a lot on this month.

Repo will be here: GitHub - penguinpowernz/mastocritters: Track US politician stock trades in Mastodon

Last years project: [Devember 2021] [COMPLETE] A telegram bot that notifies you of congress critter stock trades - #33 by penguinpowernz

HACKATHON OVER

It is mostly complete at this URL: https://indoors.trade

Things I did not finish, which were important goals (making this project a failure):

  • updating is fragile and not idempotent
  • I didn’t test the federability of the accounts
1 Like

I got Mastodon going and have managed to load Mastodon models directly into the running application in Docker through my loading method (execing a ruby require via docker exec).

I haven’t been able to get the account avatars working, it seems like it uses some strange URL: https://localhost/system/accounts/avatars/109/732/804/348/059/380/original/pete_sessions.jpg

So although I can access the avatars at https://localhost/avatars/original/pete_sessions.jpg it is not enough to set this in the Account.first.avatar_file_name variable. It uses the PaperClip lib which I don’t have experience with, so I’ll have to grok the avatar upload code to figure out how that works.

Other changes I want to make are to include that profile text which I can pull from the BioGuides. And then have it with tags for the stocks we think they currently own and their average disclosure time. Maybe other things can go in there.

Looks like the only way is to pretend you’re doing an upload:

f = ActionDispatch::Http::UploadedFile.new(
	tempfile: File.open("/opt/mastodon/public/avatars/original/pete_sessions.jpg"),
	filename: "pete_sessions.jpg",
	type: "image/jpeg",
	head: {}
)

a = Account.find_by(username: "pete_sessions")
a.update(avatar: f)

image

Now to tweak the ruby code generator.

So I realized I could just run code in the Mastodon environment. I switched to using pre-generated JSON files to figure out what to add to the database instead of generating the code for it. Seems a lot more portable.

#!/usr/bin/env ruby

require 'json'

require "/mastodon/config/environment.rb"

bios = JSON.parse(File.read("/critters/generated/bios.json"), symbolize_names: true)
metas = JSON.parse(File.read("/critters/generated/meta.json"), symbolize_names: true)

metas.each do |username, meta|
  bio = bios[username]
  a = nil
  begin
    a = Account.find_by!(username: username)
  rescue
    a = Account.create(
      username: username,
      display_name: meta[:name],
    )
  end

  a.memorial = bio.memorial
  a.note = "Mock account of %s for %s. I typically disclose trades within %d days. I did %d trades in the last 30 days. %s" % [
    bio[:position],
    bio[:state],
    meta[:avg_disclosure_days],
    meta[:trades_last30_days],
    meta[:associated_tickers].map{|t| "#"+t }.join(" ")
  ]
  a.save
end

Then I realized that if you’re in an identical docker to the real one, you have access to the database, environment and everything. So now I have just written a script with a loop, to update everything, every night.

  update:
    image: tootsuite/mastodon:v3.5.2
    env_file: .env.production
    command: /critters/bin/update.rb
    networks:
      - external_network
      - internal_network
    depends_on:
      - db
      - redis
      # - es
    volumes:
      - ./public/system:/mastodon/public/system
      - ./avatars:/mastodon/public/avatars/original
      - ./:/critters

Looks easy at the moment. About to test it.

A real pain has been figuring out how to make the trade disclosures unique. They’re very slippery with no real way to uniquely identify each.

{
  "disclosure_year":2021,
  "disclosure_date":"10/04/2021",
  "transaction_date":"2021-09-27",
  "owner":"joint",
  "ticker":"BP",
  "asset_description":"BP plc",
  "type":"purchase",
  "amount":"$1,001 - $15,000",
  "representative":"Hon. Virginia Foxx",
  "district":"NC05",
  "ptr_link":"https://disclosures-clerk.house.gov/public_disc/ptr-pdfs/2021/20019557.pdf"
}

The URL for the disclosure PDF may be present on multiple objects, and there are cases where critters will do the same trade 4 times in one day. Those 4 object are all equal.

I settled on just dumping files, like per day or something that the script processes and deletes on success.

Also people may find this file interesting:

https://raw.githubusercontent.com/penguinpowernz/mastocritters/master/generated/meta.json

Basically the overall metadata of each critter.

{
  "abigail_spanberger": {
    "name": "Abigail Spanberger",
    "user": "abigail_spanberger",
    "associated_tickers": [
      "LHX",
      "??"
    ],
    "associated_assets": [
      " ()"
    ],
    "ptr_trades": 0,
    "proper_trades": 3,
    "total_trades": 3,
    "avg_disclosure_days": 7,
    "avg_disclosure_days_last30_days": 0,
    "avg_disclosure_days_by_months": {
      "202002": 20,
      "202106": 1,
      "202209": 2
    },
    "trades_last30_days": 0,
    "trades_per_month": {
      "202002": 1,
      "202106": 1,
      "202209": 1
    },
    "buys": {
      "202002": 1,
      "202106": 1,
      "202209": 1
    },
    "sells": {},
    "partial_sells": {},
    "exchange": {}
  }
}

I also thought it would be cool to have a couple of bot accounts that do different stuff like:

  • tally up week/month stats
  • show who bought/sold the most that week/month
  • show what tickers are spiking in activity each week/month

But thats for later.

I dunno if I’m gonna make it…

I got the site up though, like the name? :laughing: https://indoors.trade

I’m close though. Just been testing and retesting the importing code.

Non-logged in Mastodon is a bit frustrating to use. I didn’t want to have normal users associated with this, just have it as a bot instance. But it may be too annoying.

Anyway, I have all of 2022’s stock trades loaded in there now.

Oh, look who it is. https://indoors.trade/@nancy_pelosi

A few extra challenges I ran into:

  • not able to match the name from a stock trade with one from the bioguide (e.g. MIchael C Burgess vs Michael Burgess)… it could be done with smarter matching
  • the same but with the avatar filenames, some are missing, likely because of the same

Ohh look Nvidia stock activity:

1 Like