Recent Blog Posts
February 4, 2020 by Steven Ng
All-in-one software tools can be very useful, as they can consolidate multiple functions while minimizing window proliferation, etc.
For example, I recently got introduced to a nifty (albeit unfortunately named) little tool called Snowflake, which lets me have terminal windows and an SFTP file browser contained within a single window. While imperfect, I find it works better for me than using WinSCP.
Snowflake, however, is a utility. All-in-one applications in other contexts may not offer the same benefits. If you haven't already gathered it from the title, I am no longer a fan of using an all-in-one system for notes and web clips.
Capturing data is a little different than a utility in that you're accumulating data into an ever-growing bucket. And if your tool is collecting different types of data, that bucket is going to get harder and harder to navigate over time.
Yes, you can categorize and classify, but that is work. And by work, I mean time and mental effort. It then begs the question of how much time do you want to spend managing that ever-growing, ultimately unsustainable bucket of data.
I used to use Evernote for self-created notes and clippings. I thought it was great, especially the web clipper, but after years of accumulating thousands of items, I realized that all-in-one data collection was a romantic notion more than it was a practical one. In other words, I was in love with the idea, but not the execution.
This realization didn't really materialize until I spent time hunting for alternatives to Evernote. It wasn't so much that there are not good alternatives to Evernote. There are plenty of good alternatives to Evernote, such as OneNote and Joplin.
The problem is that Evernote and its ilk are simply not a good model of data collection for me. I found that mixing in my notes with clipped content in the same tool just created a mountain of data that got hard to sift through. And with that realization, I decided to separate out my note taking and web clipping requirements into two discrete applications.
For notes these days, I basically just use a folder in Dropbox that contains a combination of markdown and Dropbox Paper documents.
For clipping, I use Wallabag, which is similar to Instapaper.
I have found this segregation of data to be very useful in terms of my daily workflow. That Wallabag will scrape URLs (it is mostly successful in its attempts) is also handy in that I don't have to worry about broken links in the distant future.
By eliminating clips from my notes, it's now much easier to find what I'm looking for, and it's been working very well for me.
Of course, I need to add the obligatory "your mileage may vary" disclaimer, but if you're finding yourself struggling with information overload because you're using an all-in-one data collection tool, maybe the best fix is to break up your solution into multiple tools.
January 29, 2020 by Steven Ng
Confessions of a Tab Hoarder
I've been using Edge Chromium as my primary browser on Windows for quite some time now, and I like it a lot. In the past few months, I don't think I've opened up Chrome on Windows except by mistake. Honestly, I don't miss Chrome at all.
I've always had a bad habit of having too many tabs open (often more than 50). More often than not, instead of trying to find an existing tab, I'll just open a duplicate, which exacerbates my problem.
Roll Your Own PWA
Edge has a very cool feature that lets you roll your own PWA (progressive web application) out of a web site. In other words, it will create a single-site application that is accessible from the Windows Start menu. It also means that the generated PWA gets its own spot on your taskbar, using the site's favicon as the icon. No more tab hunting.
For a tab hoarder like me, this is a life changing feature. I have a ton of web apps that I'm constantly using throughout the day, so this makes life a lot easier.
As always, there's a catch. The feature doesn't work great with all sites/web apps. If the web app you're using has to proliferate new tabs or windows, it's not a great candidate, as those tabs/windows will open up in Edge. For example, if you're thinking this technique is a great way to use Dropbox Paper or Google Apps1, you're wrong.
On the other hand, if you're using it for something like Feedly, Github or PGAdmin4, it's great. I have a lot of containerized web apps like PiHole that work great as PWAs.
So how do you enable this magical feature?
If you're running Edge for Chromium, just browse to a site (ideally, you go to the exact page you want to start from, since that will be the page you see when launching the PWA), go to the menu,
Install this site as an app. Give the app a new name if the default isn't to your liking, and bam, you're done!
- If you're looking for a way to handle multiple Google accounts for email and calendaring, check out Easymail for Gmail. I bought a license during the 1.x days, and it was fine but not something I could use daily. The latest release, 2.x, is a complete rewrite, and works much better than the previous iteration. Full disclosure, I wasn't paid to say that.
January 23, 2020 by Steven Ng
Note Taking Hell
It's no secret that I've been hunting for the "perfect" note taking application for years now.
I dropped Evernote a few years ago as a multi-purpose tool, and have been trying to find a replacement ever since.
Note taking is a very personal in that you develop some very specific tool and/or process preferences that you don't really want to compromise on. What works for one person may not work for another.
My Own Criteria
For me, my core requirements, in no particular order (they are numbered for reference), are/were:
- A Windows native desktop client
- The ability to have my notes in a hierarchical file structure that was available in a tree view (more important on the desktop than mobile)
- Cloud sync or storage
- Live conversion from Markdown to formatted text without having a preview pane
- Ability to access and read my notes from my Android phone. Editing and creating is only a nice to have, since I'm pretty much useless with a touchscreen keyboard.
- The occasional ability to collaborate on notes/docs with others (this one's not a deal breaker)
Criteria that I thought were important but ended up being unimportant:
I always found that web clipping polluted my notes a little, and I recently switched to using Wallabag for all my bookmarks and web clippings. Wallabag is very similar to Pocket, and it's been working well for me so far.
For years, I've had this crazy notion that I wanted an electronic version of the paper notebook that I bring to meetings. The problem is that the available solutions work better in my imagination than they do in real life. I've pretty much given up on the idea of a stylus-based tool, and no longer consider it meaningful criteria in determining what type of note taking tool(s) I use.
Options I Considered
With my criteria considered, some picks I tried and considered were:
The one option that did look good to me was Bear.app, but it's really only for people in the Apple ecosystem, which I'm no longer a part of.
Even if they do eventually support Windows and Android, I don't know that I'd switch to Bear, given my experience with applications that are Apple-centric. For example, I've always felt like the Windows and Android clients for 1Password were second-class citizens in terms of the product line.
A folder full of Markdown files
One of the first things I tried was a Dropbox folder full of notes using a text editor like Sublime Text. This method wasn't horrible, but the problem is that I have multiple text editors open at any given time, and it got hard finding the right window. Having said that, the solution I'm using now has the same problem, except within a web browser.
Requirements satisfied: 1, 2, 3, 5
I have a love/hate relationship with OneNote. On one hand, it's very powerful, on the other hand, it has so many user interface quirks that drive me bonkers. It's also not really a Markdown centric tool, even though it has a few Markdown-like shortcuts, like using asterisks to start bullet lists.
Requirements satisfied: 1, 2, 3, 4 (limited), 5, 6
Google Docs is fine, but the experience feels more like using a traditional word processor than a note taking tool. Also, it's not Markdown centric.
Requirements satisfied: 2, 3, 4 (limited), 5, 6
Joplin is a Markdown centric alternative to Evernote. It lets you sync your notes using Dropbox. It's pretty good, and has a web clipper that works quite well. I think the main weakness of Joplin is that it it doesn’t do live Markdown conversion the way I’d like it to.
Requirements satisfied: 1, 2, 3, 5
WikiJS is a self-hosted wiki. It has a lot of potential, but as of the time of writing, it's not quite there yet for me to want to use it daily. To be able to use it outside of my LAN, however, I'd have to open up a firewall port, or host it on a server somewhere, which is something I’m not interested in doing.
Requirements satisfied: 2, 3, 5, 6
Bookstack must be the most recommended note taking tool on /r/selfhosted. I gave it a shot, bailed, and then gave it a shot again. For note taking, it had way too many quirks with respect to the user interface that just annoyed the hell out of me. If, however, I was authoring a book, it would probably be my first choice for that task.
Requirements satisfied: 2, 3, 5, 6
Dokuwiki is a simple file based Wiki. It's ok, but I struggled with the idea of using it every day. It just wasn't very enjoyable to use.
Requirements satisfied: 2, 3, 5, 6
MarkText is similar to Bear, but lacking much of Bear's polish. It has a lot of potential, but it's still too buggy and quirky at the time of writing.
Requirements satisfied: 1, 2, 3, 4, 5
No Clear Winners
So there is obviously no tool that checks all of my 6 main requirements completely. Short of writing my own tool (which I considered, but decided I had other projects more worthy of my time), I had to give up on 100% coverage of my requirements. The solution I ultimately choose would obviously require me to compromise on what I want.
I tried Paper way back when Dropbox announced it, but honestly, it left no lasting impression on me.
I gave it another cursory shot a few weeks ago (after upgrading to a paid Dropbox subscription), and discovered that its live Markdown conversion is excellent. It is definitely better than MarkText in this regard.
It wasn't until I had to collaborate on a server installation document that I was truly sold on Paper. Before I get into the nuts and bolts of Paper, the requirements it satisfies are 2, 3, 4, 5, 6. Right now, there's no desktop client that's been released to the public. I'm hoping they release something, even if it's just an Electron app. I need something that occupies its own button on my taskbar instead of having to cycle through my browser windows to find my Notes. Having said that, in daily use, I am finding that Paper is a fantastic note taking tool.
The live conversion of Markdown works very well in Paper. The reason why Markdown support is so important to me is that it saves me a lot of time. When I'm writing documentation, the ability to style documents using markup alone is incredibly powerful. There is no mousing around for styling or setting heading levels. I find pumping out documentation to be a noticeably faster and smoother experience.
So not only do I now use Paper for notes, I also use it for work-related documents.
Where I can, I'm actually writing documentation in Paper and exporting to Word or PDF. I can say without any reservations that the editing experience in Paper is the main driver for it being my daily writing tool, in spite of any glaring weaknesses it may have.
Other Useful Features and Details
The checklist feature in Dropbox is pretty darned cool. You can assign dates and people to checklist items, and get notifications too. Paper can also aggregate the checklists on all your Paper documents into a single view.
The implementation of this feature has me rethinking how I approach project management tools in general. I still have to do some practical experiments with Paper in this regard, but I have a feeling that it can do what I need, which is saying a lot.
There's a Timeline feature that lets you insert a horizontal timeline, which can be used to create a pseudo Gantt view. This could be a great feature for collaborating on larger projects.
Like Word and Google Docs, Paper has popular features like commenting, version history, and live collaboration.
It may sound corny, but the default stylesheet for the document is really nice. Not Bear nice, but a close second. I absolutely hate the default stylesheets in Word, Onenote and most other tools. When writing long documents, you get a hiearchy map on the right side of the Paper document to allow you to jump quickly to headings in your document.
You can export your document to Word for finalization, or if you don't need to make any edits, you can export your document to PDF.
There's a presentation mode for your document, but I haven't found a good use for that feature.
While I am really liking Paper a lot, the user experience isn't all sunshine and rainbows. If the editing experience were not so good, I would consider some of these deficiencies to be deal breakers:
- The Android app experience is horrible. You can't dig for a file based on location. It's basically an "all" or "favorites" view. This is fixable, but I have to admit, it is very annoying.
- No desktop application. At any given moment, I have way too many browser windows open, each having too many tabs. Finding the window containing a Paper document is an exercise in patience. If I could at least have a dedicated Paper button in my task bar, that would save me a lot of stress.
- Paper documents are server-side documents. They are not real documents like Word or text documents. In your local Dropbox folders, the files are basically links to the Paper web app. If you have any hangups about having readable local copies, you're going to absolutely hate this about Paper. If your Internet goes out or the Paper service goes down, you are f-ed. I think a Desktop app that can have local copies would mitigate any concerns related to this.
- No easy conversion feature for existing Markdown documents. I couldn't find any way to convert my existing Markdown notes to paper short of opening the file, pasting it a new Paper document and deleting the Markdown file.
As I mentioned earlier, whether Paper is ideal for you depends on your own preferences. When it came out in 2015, I thought very little to nothing about it. I was relatively unimpressed.
Years later, after giving it a new attempt, I find the application to be a revelation. It has improved so much that it has quickly become my writing tool of choice. In fact, the draft of this blog was written in Paper and pasted into my static site generator.
Paper may not be for you, but if you care about using Markdown for writing, you should definitely give it another shot. You might be pleasantly surprised at how good it is.
January 10, 2020 by Steven Ng
Basecamp published their guide to internal communication recently, and it is insightful and excellent.
Meetings are the last resort, not the first option.
If your words can be perceived in different ways, they'll be understood in the way which does the most harm.
"Now" is often the wrong time to say what just popped into your head. It's better to let it filter it through the sieve of time. What's left is the part worth saying.
Write at the right time. Sharing something at 5pm may keep someone at work longer. You may have some spare time on a Sunday afternoon to write something, but putting it out there on Sunday may pull people back into work on the weekends. Early Monday morning communication may be buried by other things. There may not be a perfect time, but there's certainly a wrong time. Keep that in mind when you hit send.
I highly recommend that you give the article a read.
January 6, 2020 by Steven Ng
As mentioned in my first post, I have an Articles section on the site where I'll be posting evergreen articles that will continue to be updated over time.
I have two articles in place now:
- The Windows Software I use to Stay Productive - This is a list of software that I recommend for Windows users.
- My Favorite Self-Hosted Applications - This is a list of mostly Docker containers (for now) that I recommend running in your own environment.
I have a queue of other useful articles on their way, and will announce their creation or updates here on the blog.
January 6, 2020 by Steven Ng
I had to archive an Outlook mailbox to a PST file recently, but I'm not a huge fan of using Outlook for finding e-mails.
I looked around for tools that would convert a PST to JSON so I could load my data into something like Postgresql, but I didn't find anything that was free (I only need to do the conversion once).
I ended up searching for an NPM that could extract objects from a PST, and found epfromer's pst-extractor..
Using that NPM, I made a simple command line tool for Windows that can extract a PST into a JSON file.
Since my requirements were modest (one time use, reasonably sized PST file), I didn't really design it to scale. It doesn't support password protected PSTs, and I can't guarantee that it will run on a giant PST file if your machine doesn't have enough memory, but it's there if you need the functionality.
To use it, just download the executable from Github.
The command structure is pretty straightforward. From the directory of the executable, use:
pst_to_json -s /path/to/file.pst -d /path/to/output.json
There are some additional switches, but the above command is all you need to accomplish the task.
December 30, 2019 by Steven Ng
I've been working on making Docker containers of my personal Nodejs projects lately, and for those that rely on a relational database (namely Postgresql), I wanted to be able to run database migrations on them locally to make updates easier. To be clear, I'm not talking about data migrations, which are a completely different thing.
For database migrations outside of Ruby on Rails, I've been using Liquibase, and more recently, Flyway. They're both perfectly fine tools, but they also require Java, which means one may need to consider the implications of how Java licenses work.
As it relates to migrations, I subscribe to the forward migrations only philosophy, which means my requirements are relatively simple. All I need to do is scan a pre-specified directory of SQL files and run only the new SQL files. The onus is on me for providing clean, error-free SQL files, so I don't need to make a migration tool very smart.
I did look at existing libraries on NPMJS, but I didn't see any having a workflow similar to Flyway, which has been my tool of choice.
You might be getting the impression that I suffer from NIH (Not Invented Here) syndrome, and you could be right. But my experience with some software is that there are always some elements that I find incredibly frustrating or incompatible with the way I work, and I end up spending significantly more time working around those issues than I would have if I had just written something from scratch.
My NPM, named pg-forward-migrations is designed to do one very specific task well, and took a few hours to write. I spent more time figuring out how to use Liquibase the first time I used it, so in this particular case, the NIH concern is a wash.
All this NPM does is run sequence of SQL statements against a Postgresql database. The workflow is inspired by Flyway in that it uses an easy-to-remember naming convention, and it tracks completed migrations by writing to a migration table in the target database. That's it. It's not rocket science, nor is it attempting to be a replacement for a multi-platform, enterprise-class database migration tool.
At some point, I will probably use this NPM to create command line executable, but at the moment, it's not a priority.
In any case, if you've got a need for a forward migration tool for Postgresql/Node, give the NPM a whirl.
December 29, 2019 by Steven Ng
Well it's been a while since the site has been updated. A long while, actually.
It's been a long time coming, but I forced myself over the holidays to revamp the site, and here it is.
Things are going to be rough for a little while, as this is a work in progress, and I've opted to go live with a buggy site as opposed to leaving a placeholder page up. This should force me to iterate more quickly and more often.
So before I rebuilt the site, I went through a lot of hand-wringing deciding how to publish this site.
Should I use a static generator like Jekyll? A GUI type tool like MarsEdit? Should I write my own tool? Wordpress (no f-ing way)? A hosted site like Squarespace?
I ended up choosing a hodgepodge of Svelte and Sapper, with some preprocessing using Gulp. I've already been immersing myself in Svelte for the past couple of months, so I figured I may as well stick with that for my site too. Once the site is somewhat stable, I'll try to write up a blog post on what I do to build this site. TLDR; I basically write my content as markdown posts, and have a script to process them into something consumable for my Svelte app.
The site's main focus is the blog, but I will have some evergreen content in the form of articles that can help boost your productivity. I plan to publish my list of "must have" software and services, among other things.
The blog will be mostly made up of a hodgepodge of technical topics, ranging from Cognos, Node, Database and other topics that I find interesting.
In the meantime, click on the RSS link at the top of the page to keep up with my new content.