Greasemonkey Script Recovery: getting the Great Monkey back on its feet

You just lost all your Greasemonkey scripts? And the complete control of your browser, painfully built over several days of work? Looking all over the place for a Greasemonkey recovery technique posted by some tech magician?

Don't worry, I have been there too.

Damn you, BSOD! You Ate My Monkey! 😱

My OS - Windows 10 - crashed (that infamous ":(" BSOD) and restarted itself, and when I was back, I could see only a blank square where the Great Monkey used to be:

My natural instinct was thinking that my copy of GM had outlived its max life (I have a notorious track record of running with obsolete extensions); so I just went ahead and updated the extension (all the way from April 2018).

And then:

Well said. That's all I needed to hear.

Since I'm running Firefox Nightly, I didn't have the luxury of putting the blame on someone else:

Nightly is experimental and may be unstable. - Dear ol' Mozilla

No Greasemonkey. Helpless. Powerless. In a world of criminals... 🐱‍💻

So, Knight Rider quotes aside, what are my options? Rewrite all my life's work?

Naah. Not yet.

The scripts are probably still there; for whatever reason, GM cannot read them now.

Hunting for the lost GM scripts

Earlier, GM scipts were being saved in plaintext somewhere inside the Firefox user profile directory; but not anymore, it seems.

So I picked up the 61413404gyreekansoem.sqlite file from {FireFox profile home}/storage/default/moz-extension+++d2345083-0b49-4052-ac13-2cefd89be9b5/idb/, opened it quick-n-dirty in a text editor, and searched for some of my old script fragments.

Phew, they are there! But they seem to be fragmented; possibly because it's SQLite.

sqlite3.exe to the rescue

So I fired up Android SDK's SQLite tool (I didn't have anything else lying around), and loaded the file:

sqlite> .open 'C:\Users\janaka\AppData\Roaming\Mozilla\Firefox\Profiles\a4And0w1D.default\storage\default\moz-extension+++d2345083-0b49-4052-ac13-2cefd89be9b5\idb\61413404gyreekansoem.sqlite'

sqlite> .dump

PRAGMA foreign_keys=OFF;
BEGIN TRANSACTION;
CREATE TABLE database( name TEXT PRIMARY KEY, ...
...
INSERT INTO object_data VALUES(1,X'30313137653167...00001300ffff');
INSERT INTO object_data VALUES(1,X'30313965386367...00001300ffff');
...

Looks good! So why can't GM read this?

Probably their data model changed so dramatically so that my DB file no longer makes any sense?

Let's reboot Greasemonkey ⚡

FF browser console was also spewing out several errors at load time; I didn't bother delving into the sources, but it looked like relaunching GM with the existing DB was a no-go.

So I moved the DB file to a safe location, deleted the whole extension data directory (C:\Users\janaka\AppData\Roaming\Mozilla\Firefox\Profiles\a4And0w1D.default\storage\default\moz-extension+++d2345083-0b49-4052-ac13-2cefd89be9b5\) and restarted FF.

(Correction: I did back up the whole extension data directory, but it is probably not necessary.)

Nice, the Great Monkey is back!

FF also opened up the Greasemonkey startup page indicating that it treated the whole thing as a clean install. The extension data directory was also back, with a fresh DB file.

Greasemonkey script migration: first attempt

So, the million-dollar question: how the heck am I gonna load my scripts back to the new DB?

First, let's see if the schemas have actually changed:

Old DB:

CREATE TABLE database( name TEXT PRIMARY KEY, origin TEXT NOT NULL, version INTEGER NOT NULL DEFAULT 0, last_vacuum_time INTEGER NOT NULL DEFAULT 0, last_analyze_time INTEGER NOT NULL DEFAULT 0, last_vacuum_size INTEGER NOT NULL DEFAULT 0) WITHOUT ROWID;
CREATE TABLE object_store( id INTEGER PRIMARY KEY, auto_increment INTEGER NOT NULL DEFAULT 0, name TEXT NOT NULL, key_path TEXT);
CREATE TABLE object_store_index( id INTEGER PRIMARY KEY, object_store_id INTEGER NOT NULL, name TEXT NOT NULL, key_path TEXT NOT NULL, unique_index INTEGER NOT NULL, multientry INTEGER NOT NULL, locale TEXT, is_auto_locale BOOLEAN NOT NULL, FOREIGN KEY (object_store_id) REFERENCES object_store(id) );

// skimmin...

CREATE INDEX index_data_value_locale_index ON index_data (index_id, value_locale, object_data_key, value) WHERE value_locale IS NOT NULL;
CREATE TABLE unique_index_data( index_id INTEGER NOT NULL, value BLOB NOT NULL, object_store_id INTEGER NOT NULL, object_data_key BLOB NOT NULL, value_locale BLOB, PRIMARY KEY (index_id, value), FOREIGN KEY (index_id) REFERENCES object_store_index(id) , FOREIGN KEY (object_store_id, object_data_key) REFERENCES object_data(object_store_id, key) ) WITHOUT ROWID;

// skim...

CREATE TRIGGER object_data_delete_trigger AFTER DELETE ON object_data FOR EACH ROW WHEN OLD.file_ids IS NOT NULL BEGIN SELECT update_refcount(OLD.file_ids, NULL); END;
CREATE TRIGGER file_update_trigger AFTER UPDATE ON file FOR EACH ROW WHEN NEW.refcount = 0 BEGIN DELETE FROM file WHERE id = OLD.id; END;

New DB:

CREATE TABLE database( name TEXT PRIMARY KEY, origin TEXT NOT NULL, version INTEGER NOT NULL DEFAULT 0, last_vacuum_time INTEGER NOT NULL DEFAULT 0, last_analyze_time INTEGER NOT NULL DEFAULT 0, last_vacuum_size INTEGER NOT NULL DEFAULT 0) WITHOUT ROWID;
CREATE TABLE object_store( id INTEGER PRIMARY KEY, auto_increment INTEGER NOT NULL DEFAULT 0, name TEXT NOT NULL, key_path TEXT);

// skim...

CREATE TABLE unique_index_data( index_id INTEGER NOT NULL, value BLOB NOT NULL, object_store_id INTEGER NOT NULL, object_data_key BLOB NOT NULL, value_locale BLOB, PRIMARY KEY (index_id, value), FOREIGN KEY (index_id) REFERENCES object_store_index(id) , FOREIGN KEY (object_store_id, object_data_key) REFERENCES object_data(object_store_id, key) ) WITHOUT ROWID;

// skim...

CREATE TRIGGER object_data_delete_trigger AFTER DELETE ON object_data FOR EACH ROW WHEN OLD.file_ids IS NOT NULL BEGIN SELECT update_refcount(OLD.file_ids, NULL); END;
CREATE TRIGGER file_update_trigger AFTER UPDATE ON file FOR EACH ROW WHEN NEW.refcount = 0 BEGIN DELETE FROM file WHERE id = OLD.id; END;

Well, looks like both are identical!

So theoretically, I should be able to dump data from the old DB (as SQL) and insert 'em to the new DB.

no such function: update_refcount? WTF?

Old DB:

sqlite> .dump

PRAGMA foreign_keys=OFF;
...
INSERT INTO object_data VALUES(1,X'3031...ffff');
...

New DB:

sqlite> INSERT INTO object_data VALUES(1,X'3031...ffff');

Error: no such function: update_refcount

Bummer.

Is FF actually using non-standard SQLite? I didn't wait to find out.

Greasemonkey Import/Export, to the rescue!

If you haven't seen or used it before, GM comes with two handy Export a backup and Import a backup options; they basically allow you to export all your scripts as a zip file, and later import them back.

And, from earlier, you would have guessed that our "barrier to entry" is most probably the SQLite trigger statement for AFTER INSERT ON object_data:

CREATE TRIGGER object_data_insert_trigger AFTER INSERT ON object_data FOR EACH ROW WHEN NEW.file_ids IS NOT NULL BEGIN SELECT update_refcount(NULL, NEW.file_ids); END;

So, what if I:

• drop that trigger,
• import the old data dump into the new DB,
• fire up the fox,
• use Greasemonkey's Export a backup feature to export (download) the recovered scripts,
• shut down the fox,
• delete the new DB,
• restart the fox (so it creates a fresh DB), and
• import the zip file back-up?

Yay! that worked! 🎉

The second step caused me a bit of confusion. I was copying the dumped data right off the SQLite screen and pasting it into the SQLite console of the new DB; but it appeared that some lines were too long for the console input, and those inserts were causing cascading failures. Hard to figure out when you copy a huge chunk with several inserts, paste and run them, and all - except the first few - fail miserably.

So I created a SQL file out of the dumped data, and imported the file via the .read command:

Old DB:

sqlite> .output 'C:\Windows\Temp\dump.sql'
sqlite> .dump
sqlite> .quit

Cleanup:

Now, open dump.sql and remove the DDL statements; CREATE TABLE, CREATE INDEX, CREATE TRIGGER, etc.

New DB:

sqlite> DROP TRIGGER object_data_insert_trigger;
DROP TRIGGER object_data_update_trigger;
DROP TRIGGER object_data_delete_trigger;
DROP TRIGGER file_update_trigger;

sqlite> .read 'C:\Windows\Temp\dump.sql'

All else went exactly as planned.

Welcome back, O Great Monkey! 🙏

So, my Greasemonkey set up is back online, running as smoothly as ever!

Final words: if you ever hit an issue with GM:

• don't panic.
• back up your extension data directory right away.
• start from scratch (get the monkey running), and
• carefully feed the monkey (yeah, Greasemonkey) with the old configs again.