It is hard to give up airtime for any DJ, but Anjali is showing incredible solidarity with workers at XRAY.FM by pulling their (amazing) show. Anjali and The Kid will be missed on air, as will the other XRAY shows that have gone silent wweek.com/music/2021/03/…
{
"type": "entry",
"published": "2021-03-02T17:59:41+00:00",
"url": "https://twitter.com/anomalily/status/1366810455587479553",
"content": {
"text": "It is hard to give up airtime for any DJ, but Anjali is showing incredible solidarity with workers at XRAY.FM by pulling their (amazing) show. Anjali and The Kid will be missed on air, as will the other XRAY shows that have gone silent wweek.com/music/2021/03/\u2026",
"html": "It is hard to give up airtime for any DJ, but Anjali is showing incredible solidarity with workers at <a href=\"http://XRAY.FM\">XRAY.FM</a> by pulling their (amazing) show. Anjali and The Kid will be missed on air, as will the other XRAY shows that have gone silent <a href=\"https://www.wweek.com/music/2021/03/01/longtime-portland-djs-anjali-and-the-incredible-kid-have-resigned-from-xray-fm-due-to-concerns-about-the-stations-former-executive-director-jefferson-smith/\">wweek.com/music/2021/03/\u2026</a>"
},
"author": {
"type": "card",
"name": "Lillian Karabaic \ud83e\udd44\ud83c\udff3\ufe0f\u200d\ud83c\udf08 BLM",
"url": "https://twitter.com/anomalily",
"photo": "https://pbs.twimg.com/profile_images/1123802400731664385/dsHQG1nZ.jpg"
},
"post-type": "note",
"_id": "18755452",
"_source": "2773"
}
As a fan on @Patreon and a content creator (not currently active there), I was *shocked* to discover they effectively boot you off the platform the moment there’s a tiny billing snafu. Creators literally vanished before my eyes in the iPhone app. WTF?!
jaredwhite.com/podcast/67/
{
"type": "entry",
"published": "2021-03-02T17:14:25+00:00",
"url": "https://twitter.com/jaredcwhite/status/1366799065783042049",
"content": {
"text": "As a fan on @Patreon and a content creator (not currently active there), I was *shocked* to discover they effectively boot you off the platform the moment there\u2019s a tiny billing snafu. Creators literally vanished before my eyes in the iPhone app. WTF?!\n\njaredwhite.com/podcast/67/",
"html": "As a fan on <a href=\"https://twitter.com/Patreon\">@Patreon</a> and a content creator (not currently active there), I was *shocked* to discover they effectively boot you off the platform the moment there\u2019s a tiny billing snafu. Creators literally vanished before my eyes in the iPhone app. WTF?!\n\n<a href=\"https://jaredwhite.com/podcast/67/\">jaredwhite.com/podcast/67/</a>"
},
"author": {
"type": "card",
"name": "Jared White",
"url": "https://twitter.com/jaredcwhite",
"photo": "https://pbs.twimg.com/profile_images/1349070767032438784/Pns0N1s_.jpg"
},
"post-type": "note",
"_id": "18754101",
"_source": "2773"
}
Anyone have advice/experience on doing married filing jointly withholding without joint finances? (Especially with unequal income).
{
"type": "entry",
"published": "2021-03-02T16:05:21+00:00",
"url": "https://twitter.com/anomalily/status/1366781682712387584",
"content": {
"text": "Anyone have advice/experience on doing married filing jointly withholding without joint finances? (Especially with unequal income)."
},
"author": {
"type": "card",
"name": "Lillian Karabaic \ud83e\udd44\ud83c\udff3\ufe0f\u200d\ud83c\udf08 BLM",
"url": "https://twitter.com/anomalily",
"photo": "https://pbs.twimg.com/profile_images/1123802400731664385/dsHQG1nZ.jpg"
},
"post-type": "note",
"_id": "18752763",
"_source": "2773"
}
Remembering the weather app I made during my first winter after moving from Boston to Portland, OR
I made my first Android app. It reminds me once a day how much worse the weather is in Boston. It's just for me.
{
"type": "entry",
"published": "2021-03-02T15:48:45+00:00",
"url": "https://twitter.com/tinysubversions/status/1366777504514547713",
"quotation-of": "https://twitter.com/tinysubversions/status/687403272030793729",
"content": {
"text": "Remembering the weather app I made during my first winter after moving from Boston to Portland, OR"
},
"author": {
"type": "card",
"name": "Darius Kazemi",
"url": "https://twitter.com/tinysubversions",
"photo": "https://pbs.twimg.com/profile_images/1355952735921692673/XVIen_1n.jpg"
},
"post-type": "note",
"refs": {
"https://twitter.com/tinysubversions/status/687403272030793729": {
"type": "entry",
"published": "2016-01-13T22:38:06+00:00",
"url": "https://twitter.com/tinysubversions/status/687403272030793729",
"photo": [
"https://pbs.twimg.com/media/CYolnbhW8AEXDXD.png",
"https://pbs.twimg.com/media/CYoln8fWYAEOUci.png"
],
"content": {
"text": "I made my first Android app. It reminds me once a day how much worse the weather is in Boston. It's just for me."
},
"author": {
"type": "card",
"name": "Darius Kazemi",
"url": "https://twitter.com/tinysubversions",
"photo": "https://pbs.twimg.com/profile_images/1355952735921692673/XVIen_1n.jpg"
},
"post-type": "photo"
}
},
"_id": "18752204",
"_source": "2773"
}
Are you occupied by the outside of things during eternal Caturday?
{
"type": "entry",
"published": "2021-03-02T09:52:12-0500",
"url": "https://martymcgui.re/2021/03/02/095212/",
"category": [
"caturday"
],
"video": [
"https://media.martymcgui.re/9a/0d/88/16/9ae4989520eff349d3534f542060a68de8abb845ae4a967250954917.mov"
],
"content": {
"text": "Are you occupied by the outside of things during eternal Caturday?",
"html": "<p>Are you occupied by the outside of things during eternal Caturday?</p>"
},
"author": {
"type": "card",
"name": "Marty McGuire",
"url": "https://martymcgui.re/",
"photo": "https://martymcgui.re/images/logo.jpg"
},
"post-type": "video",
"_id": "18751574",
"_source": "175"
}
This is, to coin a phrase, a big fucking deal. These guys can’t stand each other.
nytimes.com/live/2021/03/0…
{
"type": "entry",
"published": "2021-03-02T15:04:09+00:00",
"url": "https://twitter.com/karabaic/status/1366766281672720391",
"content": {
"text": "This is, to coin a phrase, a big fucking deal. These guys can\u2019t stand each other.\n\nnytimes.com/live/2021/03/0\u2026",
"html": "This is, to coin a phrase, a big fucking deal. These guys can\u2019t stand each other.\n\n<a href=\"https://www.nytimes.com/live/2021/03/02/world/covid-19-coronavirus/under-a-white-house-brokered-deal-merck-will-help-boost-supplies-of-its-rival-johnson-johnsons-vaccine\">nytimes.com/live/2021/03/0\u2026</a>"
},
"author": {
"type": "card",
"name": "super duper follow",
"url": "https://twitter.com/karabaic",
"photo": "https://pbs.twimg.com/profile_images/1256785873384861696/QfRzUvne.jpg"
},
"post-type": "note",
"_id": "18751391",
"_source": "2773"
}
As open standard web components (probably with a layer of *light* tooling) increasingly take on all the duties of "views" in client-side MVC, we'll need opinionated controller/model frameworks which are content to stay in their lane. Leave the DOM alone, please and thank you!
{
"type": "entry",
"published": "2021-03-02T14:56:57+00:00",
"url": "https://twitter.com/jaredcwhite/status/1366764469796282373",
"content": {
"text": "As open standard web components (probably with a layer of *light* tooling) increasingly take on all the duties of \"views\" in client-side MVC, we'll need opinionated controller/model frameworks which are content to stay in their lane. Leave the DOM alone, please and thank you!"
},
"author": {
"type": "card",
"name": "Jared White",
"url": "https://twitter.com/jaredcwhite",
"photo": "https://pbs.twimg.com/profile_images/1349070767032438784/Pns0N1s_.jpg"
},
"post-type": "note",
"_id": "18751101",
"_source": "2773"
}
The Glitch union has signed its collective bargaining agreement with the company — the first time this has happened with white collar workers in the tech industry. Members say the company’s decision to voluntarily recognize the union made contract negotiations amicable:
Glitch workers sign tech’s first collective bargaining agreement theverge.com/2021/3/2/22307…
{
"type": "entry",
"published": "2021-03-02T14:22:59+00:00",
"url": "https://twitter.com/ZoeSchiffer/status/1366755923654418432",
"quotation-of": "https://twitter.com/verge/status/1366750168431947780",
"content": {
"text": "The Glitch union has signed its collective bargaining agreement with the company \u2014 the first time this has happened with white collar workers in the tech industry. Members say the company\u2019s decision to voluntarily recognize the union made contract negotiations amicable:"
},
"author": {
"type": "card",
"name": "Zo\u00eb Schiffer",
"url": "https://twitter.com/ZoeSchiffer",
"photo": "https://pbs.twimg.com/profile_images/1339414834350182401/Cbwg03OA.jpg"
},
"post-type": "note",
"refs": {
"https://twitter.com/verge/status/1366750168431947780": {
"type": "entry",
"published": "2021-03-02T14:00:07+00:00",
"url": "https://twitter.com/verge/status/1366750168431947780",
"photo": [
"https://pbs.twimg.com/media/EvesCPqXMAASNPd.jpg"
],
"content": {
"text": "Glitch workers sign tech\u2019s first collective bargaining agreement theverge.com/2021/3/2/22307\u2026",
"html": "Glitch workers sign tech\u2019s first collective bargaining agreement <a href=\"https://www.theverge.com/2021/3/2/22307671/glitch-workers-sign-historic-collective-bargaining-agreement-cwa?utm_campaign=theverge&utm_content=chorus&utm_medium=social&utm_source=twitter\">theverge.com/2021/3/2/22307\u2026</a>"
},
"author": {
"type": "card",
"name": "The Verge",
"url": "https://twitter.com/verge",
"photo": "https://pbs.twimg.com/profile_images/877903823133704194/Mqp1PXU8.jpg"
},
"post-type": "photo"
}
},
"_id": "18750395",
"_source": "2773"
}
This is my passport photo from 1991. So weird...
{
"type": "entry",
"published": "2021-03-02T12:49:45+00:00",
"url": "https://twitter.com/afitnerd/status/1366732456846630912",
"video": [
"https://video.twimg.com/ext_tw_video/1366732438085459968/pu/vid/512x512/T7ypL3puFPxGfRf-.mp4?tag=10"
],
"content": {
"text": "This is my passport photo from 1991. So weird..."
},
"author": {
"type": "card",
"name": "Micah Silverman - #BLM - Wear a Mask",
"url": "https://twitter.com/afitnerd",
"photo": "https://pbs.twimg.com/profile_images/1169746170740137984/R7RpX8Q7.jpg"
},
"post-type": "video",
"_id": "18749039",
"_source": "2773"
}
Deep nostalgia is amazing. And creepy. This is my grandmother's wedding photo, animated...
{
"type": "entry",
"published": "2021-03-02T12:48:35+00:00",
"url": "https://twitter.com/afitnerd/status/1366732164163903489",
"video": [
"https://video.twimg.com/ext_tw_video/1366732142928134144/pu/vid/512x512/f3lnjwsGRsZGtep7.mp4?tag=10"
],
"content": {
"text": "Deep nostalgia is amazing. And creepy. This is my grandmother's wedding photo, animated..."
},
"author": {
"type": "card",
"name": "Micah Silverman - #BLM - Wear a Mask",
"url": "https://twitter.com/afitnerd",
"photo": "https://pbs.twimg.com/profile_images/1169746170740137984/R7RpX8Q7.jpg"
},
"post-type": "video",
"_id": "18748734",
"_source": "2773"
}
Nice definition. I would replace “1 bit” by “more than zero bits” just to be extra pedantic.
twitter.com/alecmuffett/st…
What do we mean by a "backdoor" in End-To-End Encrypted Messengers or Secure Messengers? #endToEndEncryption #e2ee alecmuffett.com/article/14275
{
"type": "entry",
"published": "2021-03-02T08:49:09+00:00",
"url": "https://twitter.com/Johannes_Ernst/status/1366671907777511431",
"quotation-of": "https://twitter.com/AlecMuffett/status/1366667087326961664",
"content": {
"text": "Nice definition. I would replace \u201c1 bit\u201d by \u201cmore than zero bits\u201d just to be extra pedantic.\ntwitter.com/alecmuffett/st\u2026",
"html": "Nice definition. I would replace \u201c1 bit\u201d by \u201cmore than zero bits\u201d just to be extra pedantic.\n<a href=\"https://twitter.com/alecmuffett/status/1366667087326961664\">twitter.com/alecmuffett/st\u2026</a>"
},
"author": {
"type": "card",
"name": "Johannes Ernst",
"url": "https://twitter.com/Johannes_Ernst",
"photo": "https://pbs.twimg.com/profile_images/462335209015238656/ie0cRjdx.jpeg"
},
"post-type": "note",
"refs": {
"https://twitter.com/AlecMuffett/status/1366667087326961664": {
"type": "entry",
"published": "2021-03-02T08:29:59+00:00",
"url": "https://twitter.com/AlecMuffett/status/1366667087326961664",
"content": {
"text": "What do we mean by a \"backdoor\" in End-To-End Encrypted Messengers or Secure Messengers? #endToEndEncryption #e2ee alecmuffett.com/article/14275",
"html": "What do we mean by a \"backdoor\" in End-To-End Encrypted Messengers or Secure Messengers? <a href=\"https://twitter.com/search?q=%23endToEndEncryption\">#endToEndEncryption</a> <a href=\"https://twitter.com/search?q=%23e2ee\">#e2ee</a> <a href=\"https://alecmuffett.com/article/14275\">alecmuffett.com/article/14275</a>"
},
"author": {
"type": "card",
"name": "Alec Muffett",
"url": "https://twitter.com/AlecMuffett",
"photo": "https://pbs.twimg.com/profile_images/968073512731717632/HWo_YLQs.jpg"
},
"post-type": "note"
}
},
"_id": "18745190",
"_source": "2773"
}
Doing another stream for some reason. twitch.tv/fluffycritter
{
"type": "entry",
"published": "2021-03-02T04:24:00+00:00",
"url": "https://twitter.com/fluffy/status/1366605181853466626",
"content": {
"text": "Doing another stream for some reason. twitch.tv/fluffycritter",
"html": "Doing another stream for some reason. <a href=\"https://twitch.tv/fluffycritter\">twitch.tv/fluffycritter</a>"
},
"author": {
"type": "card",
"name": "fluffy \ud83d\udc9c \u272a\u25be\u032b\u272a",
"url": "https://twitter.com/fluffy",
"photo": "https://pbs.twimg.com/profile_images/1252480375646851072/Pc5osn74.jpg"
},
"post-type": "note",
"_id": "18742426",
"_source": "2773"
}
{
"type": "entry",
"published": "2021-03-02T03:53:27+00:00",
"url": "https://twitter.com/karabaic/status/1366597496487563267",
"content": {
"text": "Very cool nytimes.com/2021/02/24/sci\u2026",
"html": "Very cool <a href=\"https://www.nytimes.com/2021/02/24/science/nasa-mars-parachute-code.html?referringSource=articleShare\">nytimes.com/2021/02/24/sci\u2026</a>"
},
"author": {
"type": "card",
"name": "super duper follow",
"url": "https://twitter.com/karabaic",
"photo": "https://pbs.twimg.com/profile_images/1256785873384861696/QfRzUvne.jpg"
},
"post-type": "note",
"_id": "18742003",
"_source": "2773"
}
{
"type": "entry",
"published": "2021-03-02T01:08:06+00:00",
"url": "https://cleverdevil.io/2021/taking-control-of-my-personal-health-data",
"category": [
"indienews",
"https://news.indieweb.org/en"
],
"name": "Taking Control of my Personal Health Data",
"content": {
"text": "Over the past few years, I've invested time and effort into extricating important data and content from external services, and bringing it into systems that I own and control. I've moved on from Facebook and Instagram, established tracking for my movie, tv, and podcast activity, automatically track my location in multiple ways, and much more. But, for years now, one type of data has eluded me: my personal health data.As of today, that has changed! I'd like to share with you what I've built.Overview of EnhancementsMy website now features my personal health metrics in several places. First, there is now a health section which shows both daily health metrics and historical metrics. You can go backward and forward in time and compare my daily metrics to historical min, max, and average values.For the daily metrics, I use the familiar Apple Activity Rings format, and include supporting metrics across a variety of categories, including activity, heart health, and sleep analysis.For the historical metrics, I am particularly proud of the visualization. Each metric has a bar representing the minimum, maximum, and average values, and the gradient that is used to fill the bar adjusts to reflect the position of the average value.In addition, I have augmented my monthly summaries.Each day is represented by an Activity Ring and can be clicked on to view detailed, in-context metrics for that day.Overall, I am quite pleased with how this project has turned out. Navigating through health metrics is snappy, the visualizations are attractive and useful, and it fits in neatly with the rest of my site.Now that we've walked through what these features look like in practice, let's discuss how I gather the data and make it useful.Unlocking HealthKitI've owned an Apple Watch since the Series 2 watch was released, and have worn it fairly consistently ever since. As a result, I've got quite a lot of data amassed on my iPhone in Apple Health. That data is accessible through the Health app, and also via the HealthKit APIs. While I am a pretty strong developer, my skillset doesn't include much in the way of iOS development. I've made a few attempts at building an iOS app that will allow me to extract my HealthKit data automatically, but never made it far before I ran out of steam.A few weeks ago, I discovered an app called Health Auto Export (which I will refer to as HAE for the rest of this post), which neatly solves the problem. HAE has many great features, but the key feature is \"API Export,\" which allows you to automatically have your HealthKit data sent to an HTTP endpoint in JSON or CSV\u00a0format, with control over time period and aggregation granularity. With this app in hand, I set about creating an API to store, index, and make that data searchable.Introducing Health LakeHAE uses a simple, but nested JSON\u00a0data structure to represent health metrics. Because the data is structured, in plain-text, and will mostly sit at rest, a data lake is a natural target to store the data. Data lakes on Amazon Web Services (AWS) are generally implemented with Amazon S3 for storage, as it is well-suited to the use case, is deeply integrated with AWS' data, analytics, and machine learning (DAML) services.In order to keep most of the complexity out of my website, I decided to build a microservice which is entirely focused on getting data into the data lake and making it useful. I call this service Health Lake, and the source is available on GitHub.Sync and StoreLet's take a look at the first endpoint of Health Lake, which accepts data from HAE, trasforms it to align with the requirments for AWS's DAML services, and stores it in S3 - HTTP POST /sync.HAE structures its data in a nested format:{\n \"data\": {\n \"metrics\": [\n {\n \"units\": \"kcal\",\n \"name\": \"active_energy\",\n \"data\": [\n {\n \"date\": \"2021-01-20 00:00:00 -0800\",\n \"qty\": 370.75\n },\n ...\n ]\n },\n ...\n ],\n }\n}\nAs you can see, the data is nested fairly deeply. In order to simplify my ability to query the data, Health Lake transforms the data to a flatter structure, with each data point being formatted in JSON\u00a0on a single line. On each sync, I create a single object that contains many data points, one per line, in a format like this:{\"name\": \"active_energy\", \"date\": \"2021-01-20 00:00:00 -0800\", \"units\": \"kcal\", \"qty\": 370.75 }\n...\nEach sync object is stored in my target S3 bucket with the key format:syncs/<ISO-format date and time of sync>.jsonThe prefix on the object name is critical, as it enables the indexing and querying of sync data independent from other data in the bucket.Querying the Data LakeNow that we have data being sent to our data lake and stored in an efficient, standardized format, we can focus on making that data searchable. Very often, I use relational databases like MySQL or PostgreSQL to store data and make it searchable with SQL. AWS provides a few great services which allow you to treat your data lake as a series of database tables that can be queried using SQL.The first service we'll leverage is AWS Glue, which provides powerful data integration capabilities:\nAWS Glue is a serverless data integration service that makes it easy to discover, prepare, and combine data for analytics, machine learning, and application development. AWS Glue provides all of the capabilities needed for data integration so that you can start analyzing your data and putting it to use in minutes instead of months.\nData integration is the process of preparing and combining data for analytics, machine learning, and application development. It involves multiple tasks, such as discovering and extracting data from various sources; enriching, cleaning, normalizing, and combining data; and loading and organizing data in databases, data warehouses, and data lakes.\nUsing AWS Glue, I created a database called \"health,\" and then created a \"crawler,\" which connects to my data store in S3, walks through all of the data, and attempts to infer the schema based upon hints and classifiers. The crawler can be run manually on-demand, or can be scheduled to run on a regular basis to continuously update the schema as new fields are discovered. Here is what the configuration of my crawler looks like in the AWS Glue console:Upon the first run of the crawler, a new table was created in my health database called syncs, which inferred the following schema:I wasn't able to get the crawler to match the date format properly, so I ended up creating a \"view\" which adds a proper column that is a timestamp using the following SQL statement:CREATE OR REPLACE VIEW \n history \nAS SELECT\n date_parse(substr(date, 1, 19), '%Y-%m-%d %H:%i:%s') as datetime,\n *\nFROM\n syncs\nNow that our data lake has been crawled, and a database, table, and view have been defined in our AWS Glue Data Catalog, we can use Amazon Athena to query our data like using standard SQL. Athena is entirely serverless, so there is no infrastructure to manage, and you pay only for the queries that you run.Daily MetricsFor our daily metric view, we need a summary of all metrics gathered on a specific day. To accomplish this, I added an endpoint to our microservice:HTTP GET /detail/<YYYY-MM-DD>In response to this request, the client will receive a JSON data structure collecting all data points for that day. Under the hood, the microservice is running the following SQL query:SELECT * FROM history \nWHERE\n datetime >= TIMESTAMP 'YYYY-MM-DD 00:00:00'\nAND\n datetime <= TIMESTAMP 'YYYY-MM-DD 23:59:59'\nBecause I pay for every query that I run on Athena, and to achieve great performance, I store the query results in the proper format for the client in S3 after I run the query. I then implemented some intelligence to decide if, for any given request, I should pull from the cache, or regenerate fresh data. Take a look at the source code for more detail.Monthly MetricsTo show our monthly summaries, we need to get data for each day of the month. Rather than sending a request and query for every single day of the month, I decided to implement another endpoint to our microservice:HTTP GET /summary/<YYYY-MM>In response to this request, the client will receive a JSON data structure collecting all data points for the month, sorted by date. To accomplish this, I run the following SQL query:SELECT * FROM history\nWHERE\n datetime >= TIMESTAMP 'YYYY-MM-01 00:00:00'\nAND\n datetime <= TIMESTAMP 'YYYY-MM-31 00:00:00'\nThe start and end range are actually calculated to ensure I have the proper end date, as not every month has the same number of days. Again, to save costs and improve performance, results are intelligently cached in our S3 bucket.Global MetricsGenerating a global summary of all data points in the data lake was a bit more challenging. To make things more efficient, I created another view in my database with this query. Results are, again, intelligently cached.Website IntegrationWith all of this great data available to me, it was time to integrate it with my website, which uses the Known CMS. I have created a Known plugin that provides enhancements that are specific to my website. Using this plugin, I simply send requests to the Health Lake microservice, parse the JSON, and create my visualizations.ConclusionsOverall, I am quite pleased that I have been able to integrate this data into my website, and more importantly, to free the data from its walled garden and place it under my control and ownership.#indienews",
"html": "<p>Over the past few years, I've invested time and effort into extricating important data and content from external services, and bringing it into systems that I own and control. I've <a href=\"https://cleverdevil.io/2018/freeing-myself-from-facebook\">moved on from Facebook and Instagram</a>, <a href=\"https://cleverdevil.io/2019/tracking-my-movie-tv-and-podcast-activity\">established tracking for my movie, tv, and podcast activity</a>, automatically <a href=\"https://cleverdevil.io/content/locations/\">track my location</a> in <a href=\"https://github.com/cleverdevil/punytrack\">multiple ways</a>, and much more. But, for years now, one type of data has eluded me: my personal health data.</p><p>As of today, that has changed! I'd like to share with you what I've built.</p><h2>Overview of Enhancements</h2><p>My website now features my personal health metrics in several places. First, there is now a <a href=\"https://cleverdevil.io/health\">health section</a> which shows both daily health metrics and historical metrics. You can go backward and forward in time and compare my daily metrics to historical min, max, and average values.</p><p>For the daily metrics, I use the familiar Apple Activity Rings format, and include supporting metrics across a variety of categories, including activity, heart health, and sleep analysis.</p><p><img style=\"margin-left:auto;margin-right:auto;\" src=\"https://cleverdevil.io/file/35c4ef8c4dac4cd73a1137b4b5f3427f/thumb.png\" alt=\"Daily Health Metrics Screenshot\" width=\"600\" height=\"205\" /></p><p>For the historical metrics, I am particularly proud of the visualization. Each metric has a bar representing the minimum, maximum, and average values, and the gradient that is used to fill the bar adjusts to reflect the position of the average value.</p><p><img style=\"margin-left:auto;margin-right:auto;\" src=\"https://cleverdevil.io/file/37fd561b24daad460c1ef21419ec24c2/thumb.png\" alt=\"Historical Health Metrics Screenshot\" width=\"600\" height=\"303\" /></p><p>In addition, I have augmented <a href=\"https://cleverdevil.io/summary/2020/03\">my</a> <a href=\"https://cleverdevil.io/summary/2020/01\">monthly</a> <a href=\"https://cleverdevil.io/summary/2019/10\">summaries</a>.</p><p><img style=\"margin-left:auto;margin-right:auto;\" src=\"https://cleverdevil.io/file/3be948291f8ac5eb38f072f31fb04fc8/thumb.png\" alt=\"Monthly Health Summary Screenshot\" width=\"600\" height=\"178\" /></p><p>Each day is represented by an Activity Ring and can be clicked on to view detailed, in-context metrics for that day.</p><p>Overall, I am quite pleased with how this project has turned out. Navigating through health metrics is snappy, the visualizations are attractive and useful, and it fits in neatly with the rest of my site.</p><p>Now that we've walked through what these features look like in practice, let's discuss how I gather the data and make it useful.</p><h2>Unlocking HealthKit</h2><p>I've owned an Apple Watch since the Series 2 watch was released, and have worn it fairly consistently ever since. As a result, I've got quite a lot of data amassed on my iPhone in <a href=\"https://www.apple.com/ios/health/\">Apple Health</a>. That data is accessible through the Health app, and also via the <a href=\"https://developer.apple.com/documentation/healthkit\">HealthKit APIs</a>. While I am a pretty strong developer, my skillset doesn't include much in the way of iOS development. I've made a few attempts at building an iOS app that will allow me to extract my HealthKit data automatically, but never made it far before I ran out of steam.</p><p>A few weeks ago, I discovered an app called <a href=\"https://apps.apple.com/us/app/health-auto-export-json-csv/id1115567069\">Health Auto Export</a> (which I will refer to as HAE for the rest of this post), which neatly solves the problem. HAE has many great features, but the key feature is \"API Export,\" which allows you to automatically have your HealthKit data sent to an HTTP endpoint in JSON or CSV\u00a0format, with control over time period and aggregation granularity. With this app in hand, I set about creating an API to store, index, and make that data searchable.</p><h2>Introducing Health Lake</h2><p>HAE uses a simple, but nested JSON\u00a0data structure to represent health metrics. Because the data is structured, in plain-text, and will mostly sit at rest, a <a href=\"https://en.wikipedia.org/wiki/Data_lake\">data lake</a> is a natural target to store the data. Data lakes on Amazon Web Services (AWS) are generally implemented with <a href=\"https://aws.amazon.com/s3/\">Amazon S3</a> for storage, as it is well-suited to the use case, is deeply integrated with AWS' data, analytics, and machine learning (DAML) services.</p><p>In order to keep most of the complexity out of my website, I decided to build a microservice which is entirely focused on getting data into the data lake and making it useful. I call this service Health Lake, and <a href=\"https://github.com/cleverdevil/healthlake\">the source is available on GitHub</a>.</p><h3>Sync and Store</h3><p>Let's take a look at the first endpoint of Health Lake, which accepts data from HAE, trasforms it to align with the requirments for AWS's DAML services, and stores it in S3 - <code>HTTP POST /sync</code>.</p><p>HAE structures its data in a nested format:</p><pre><code>{\n \"data\": {\n \"metrics\": [\n {\n \"units\": \"kcal\",\n \"name\": \"active_energy\",\n \"data\": [\n {\n \"date\": \"2021-01-20 00:00:00 -0800\",\n \"qty\": 370.75\n },\n ...\n ]\n },\n ...\n ],\n }\n}\n</code></pre><p>As you can see, the data is nested fairly deeply. In order to simplify my ability to query the data, Health Lake transforms the data to a flatter structure, with each data point being formatted in JSON\u00a0on a single line. On each sync, I create a single object that contains many data points, one per line, in a format like this:</p><pre><code>{\"name\": \"active_energy\", \"date\": \"2021-01-20 00:00:00 -0800\", \"units\": \"kcal\", \"qty\": 370.75 }\n...\n</code></pre><p>Each sync object is stored in my target S3 bucket with the key format:</p><p><code>syncs/<ISO-format date and time of sync>.json</code></p><p>The prefix on the object name is critical, as it enables the indexing and querying of sync data independent from other data in the bucket.</p><h3>Querying the Data Lake</h3><p>Now that we have data being sent to our data lake and stored in an efficient, standardized format, we can focus on making that data searchable. Very often, I use relational databases like MySQL or PostgreSQL to store data and make it searchable with SQL. AWS provides a few great services which allow you to treat your data lake as a series of database tables that can be queried using SQL.</p><p>The first service we'll leverage is <a href=\"https://aws.amazon.com/glue/\">AWS Glue</a>, which provides powerful data integration capabilities:</p><blockquote>\n<p>AWS Glue is a serverless data integration service that makes it easy to discover, prepare, and combine data for analytics, machine learning, and application development. AWS Glue provides all of the capabilities needed for data integration so that you can start analyzing your data and putting it to use in minutes instead of months.</p>\n<p>Data integration is the process of preparing and combining data for analytics, machine learning, and application development. It involves multiple tasks, such as discovering and extracting data from various sources; enriching, cleaning, normalizing, and combining data; and loading and organizing data in databases, data warehouses, and data lakes.</p>\n</blockquote><p>Using AWS Glue, I created a database called \"health,\" and then created a \"crawler,\" which connects to my data store in S3, walks through all of the data, and attempts to infer the schema based upon hints and classifiers. The crawler can be run manually on-demand, or can be scheduled to run on a regular basis to continuously update the schema as new fields are discovered. Here is what the configuration of my crawler looks like in the AWS Glue console:</p><p><img style=\"margin-left:auto;margin-right:auto;\" src=\"https://cleverdevil.io/file/a3b08668d120d1b1c0ab7db1882c84b4/thumb.png\" alt=\"AWS Glue Crawler Configuration Screenshot\" width=\"600\" height=\"474\" /></p><p>Upon the first run of the crawler, a new table was created in my health database called <code>syncs</code>, which inferred the following schema:</p><p><img style=\"margin-left:auto;margin-right:auto;\" src=\"https://cleverdevil.io/file/4f3bff75ab10cdd516698d295b1aea79\" alt=\"AWS Glue Table Schema Screenshot\" width=\"300\" height=\"439\" /></p><p>I wasn't able to get the crawler to match the date format properly, so I ended up creating a \"view\" which adds a proper column that is a <code>timestamp</code> using the following SQL statement:</p><pre><code>CREATE OR REPLACE VIEW \n history \nAS SELECT\n date_parse(substr(date, 1, 19), '%Y-%m-%d %H:%i:%s') as datetime,\n *\nFROM\n syncs\n</code></pre><p>Now that our data lake has been crawled, and a database, table, and view have been defined in our AWS Glue Data Catalog, we can use <a href=\"https://aws.amazon.com/athena/\">Amazon Athena</a> to query our data like using standard SQL. Athena is entirely serverless, so there is no infrastructure to manage, and you pay only for the queries that you run.</p><h3>Daily Metrics</h3><p>For our daily metric view, we need a summary of all metrics gathered on a specific day. To accomplish this, I added an endpoint to our microservice:</p><p><code>HTTP GET /detail/<YYYY-MM-DD></code></p><p>In response to this request, the client will receive a JSON data structure collecting all data points for that day. Under the hood, the microservice is running the following SQL query:</p><pre><code>SELECT * FROM history \nWHERE\n datetime >= TIMESTAMP 'YYYY-MM-DD 00:00:00'\nAND\n datetime <= TIMESTAMP 'YYYY-MM-DD 23:59:59'\n</code></pre><p>Because I pay for every query that I run on Athena, and to achieve great performance, I store the query results in the proper format for the client in S3 after I run the query. I then implemented some intelligence to decide if, for any given request, I should pull from the cache, or regenerate fresh data. Take a look at <a href=\"https://github.com/cleverdevil/healthlake/blob/97fecaa7d651b09a11c100e672d9a3e64f6f11da/healthlake.py#L311\">the source code</a> for more detail.</p><h3>Monthly Metrics</h3><p>To show our monthly summaries, we need to get data for each day of the month. Rather than sending a request and query for every single day of the month, I decided to implement another endpoint to our microservice:</p><p><code>HTTP GET /summary/<YYYY-MM></code></p><p>In response to this request, the client will receive a JSON data structure collecting all data points for the month, sorted by date. To accomplish this, I run the following SQL query:</p><pre><code>SELECT * FROM history\nWHERE\n datetime >= TIMESTAMP 'YYYY-MM-01 00:00:00'\nAND\n datetime <= TIMESTAMP 'YYYY-MM-31 00:00:00'\n</code></pre><p>The start and end range are actually calculated to ensure I have the proper end date, as not every month has the same number of days. Again, to save costs and improve performance, results are intelligently cached in our S3 bucket.</p><h3>Global Metrics</h3><p>Generating a global summary of all data points in the data lake was a bit more challenging. To make things more efficient, I created another view in my database <a href=\"https://github.com/cleverdevil/healthlake/blob/main/schemas/global_metrics.sql\">with this query</a>. Results are, again, <a href=\"https://github.com/cleverdevil/healthlake/blob/97fecaa7d651b09a11c100e672d9a3e64f6f11da/healthlake.py#L369\">intelligently cached</a>.</p><h2>Website Integration</h2><p>With all of this great data available to me, it was time to integrate it with my website, which uses the <a href=\"https://github.com/idno/known\">Known CMS</a>. I have created <a href=\"https://github.com/cleverdevil/CleverCustomize\">a Known plugin</a> that provides enhancements that are specific to my website. Using this plugin, I simply send requests to the Health Lake microservice, parse the JSON, and create my visualizations.</p><h2>Conclusions</h2><p>Overall, I am quite pleased that I have been able to integrate this data into my website, and more importantly, to free the data from its walled garden and place it under my control and ownership.</p><p><a class=\"u-category\" href=\"https://news.indieweb.org/en\"></a><a href=\"https://cleverdevil.io/tag/indienews\" class=\"p-category\">#indienews</a></p>"
},
"author": {
"type": "card",
"name": "Jonathan LaCour",
"url": "https://cleverdevil.io/profile/cleverdevil",
"photo": "https://cleverdevil.io/file/fdbc7696a5f73864ea11a828c861e138/thumb.jpg"
},
"post-type": "article",
"_id": "18740807",
"_source": "10"
}
Paying verrry close attention to Below Deck Sailing Yacht this season as they sail around Croatia. Starting off in the south, hoping they visit where my family is from. Really want to see if distant relatives bilk the tourists.
{
"type": "entry",
"published": "2021-03-02T02:11:17+00:00",
"url": "https://twitter.com/karabaic/status/1366571783499915265",
"content": {
"text": "Paying verrry close attention to Below Deck Sailing Yacht this season as they sail around Croatia. Starting off in the south, hoping they visit where my family is from. Really want to see if distant relatives bilk the tourists."
},
"author": {
"type": "card",
"name": "super duper follow",
"url": "https://twitter.com/karabaic",
"photo": "https://pbs.twimg.com/profile_images/1256785873384861696/QfRzUvne.jpg"
},
"post-type": "note",
"_id": "18740749",
"_source": "2773"
}
Just started streaming Fights in Tight Spaces, a great game that is like Into the Breach meets Slay the Spire. But about action movie fighting.
twitch.tv/tinysubversions
{
"type": "entry",
"published": "2021-03-02T02:07:59+00:00",
"url": "https://twitter.com/tinysubversions/status/1366570954680279041",
"content": {
"text": "Just started streaming Fights in Tight Spaces, a great game that is like Into the Breach meets Slay the Spire. But about action movie fighting.\n\ntwitch.tv/tinysubversions",
"html": "Just started streaming Fights in Tight Spaces, a great game that is like Into the Breach meets Slay the Spire. But about action movie fighting.\n\n<a href=\"https://www.twitch.tv/tinysubversions\">twitch.tv/tinysubversions</a>"
},
"author": {
"type": "card",
"name": "Darius Kazemi",
"url": "https://twitter.com/tinysubversions",
"photo": "https://pbs.twimg.com/profile_images/1355952735921692673/XVIen_1n.jpg"
},
"post-type": "note",
"_id": "18740750",
"_source": "2773"
}
Starting Mario Cuomo’s political career, Italian ice, and being the epicenter of the COVID-19 pandemic about a year ago
twitter.com/markgrote/stat…
Without naming where you're from, what is it famous or known for?
{
"type": "entry",
"published": "2021-03-02T02:01:01+00:00",
"url": "https://twitter.com/karabaic/status/1366569197866606592",
"quotation-of": "https://twitter.com/MarkGrote/status/1365900243720544262",
"content": {
"text": "Starting Mario Cuomo\u2019s political career, Italian ice, and being the epicenter of the COVID-19 pandemic about a year ago\ntwitter.com/markgrote/stat\u2026",
"html": "Starting Mario Cuomo\u2019s political career, Italian ice, and being the epicenter of the COVID-19 pandemic about a year ago\n<a href=\"https://twitter.com/markgrote/status/1365900243720544262\">twitter.com/markgrote/stat\u2026</a>"
},
"author": {
"type": "card",
"name": "super duper follow",
"url": "https://twitter.com/karabaic",
"photo": "https://pbs.twimg.com/profile_images/1256785873384861696/QfRzUvne.jpg"
},
"post-type": "note",
"refs": {
"https://twitter.com/MarkGrote/status/1365900243720544262": {
"type": "entry",
"published": "2021-02-28T05:42:49+00:00",
"url": "https://twitter.com/MarkGrote/status/1365900243720544262",
"content": {
"text": "Without naming where you're from, what is it famous or known for?"
},
"author": {
"type": "card",
"name": "Mark Grote \ud83d\udde8\ufe0f",
"url": "https://twitter.com/MarkGrote",
"photo": "https://pbs.twimg.com/profile_images/1333261005271363585/VTshIdIq.jpg"
},
"post-type": "note"
}
},
"_id": "18740523",
"_source": "2773"
}
Today is a historic day: for the first time in history we have two Marches at once.
{
"type": "entry",
"published": "2021-03-02T01:37:27+00:00",
"url": "https://twitter.com/fluffy/status/1366563266927857664",
"content": {
"text": "Today is a historic day: for the first time in history we have two Marches at once."
},
"author": {
"type": "card",
"name": "fluffy \ud83d\udc9c \u272a\u25be\u032b\u272a",
"url": "https://twitter.com/fluffy",
"photo": "https://pbs.twimg.com/profile_images/1252480375646851072/Pc5osn74.jpg"
},
"post-type": "note",
"_id": "18740361",
"_source": "2773"
}
For those that are curious, I wrote up a summary of how I integrated my personal health data from Apple HealthKit into my website. - cleverdevil.io/2021/taking-co…
{
"type": "entry",
"published": "2021-03-02T01:32:28+00:00",
"url": "https://twitter.com/cleverdevil/status/1366562014395576322",
"content": {
"text": "For those that are curious, I wrote up a summary of how I integrated my personal health data from Apple HealthKit into my website. - cleverdevil.io/2021/taking-co\u2026",
"html": "For those that are curious, I wrote up a summary of how I integrated my personal health data from Apple HealthKit into my website. - <a href=\"https://cleverdevil.io/2021/taking-control-of-my-personal-health-data\">cleverdevil.io/2021/taking-co\u2026</a>"
},
"author": {
"type": "card",
"name": "Jonathan LaCour",
"url": "https://twitter.com/cleverdevil",
"photo": "https://pbs.twimg.com/profile_images/744804831064317952/W-gMo7AO.jpg"
},
"post-type": "note",
"_id": "18740203",
"_source": "2773"
}
“As you can see by looking at the nonsense math for this fantasy scenario, it’s actually good that so many people are poor and dying.”
If you start a successful startup in your twenties, a 2% wealth tax with a $50 million threshold means that over the course of your life the government will take 65% of y...
{
"type": "entry",
"published": "2021-03-02T01:03:01+00:00",
"url": "https://twitter.com/matthewmcvickar/status/1366554604935127044",
"quotation-of": "https://twitter.com/paulg/status/1366439274212700163",
"content": {
"text": "\u201cAs you can see by looking at the nonsense math for this fantasy scenario, it\u2019s actually good that so many people are poor and dying.\u201d"
},
"author": {
"type": "card",
"name": "Matthew McVickar",
"url": "https://twitter.com/matthewmcvickar",
"photo": "https://pbs.twimg.com/profile_images/1082744131414155264/Y6nU3C3a.jpg"
},
"post-type": "note",
"refs": {
"https://twitter.com/paulg/status/1366439274212700163": {
"type": "entry",
"published": "2021-03-01T17:24:44+00:00",
"url": "https://twitter.com/paulg/status/1366439274212700163",
"content": {
"text": "If you start a successful startup in your twenties, a 2% wealth tax with a $50 million threshold means that over the course of your life the government will take 65% of your stock. Why so much? A wealth tax compounds.\n\npaulgraham.com/wtax.html",
"html": "If you start a successful startup in your twenties, a 2% wealth tax with a $50 million threshold means that over the course of your life the government will take 65% of your stock. Why so much? A wealth tax compounds.\n\n<a href=\"http://paulgraham.com/wtax.html\">paulgraham.com/wtax.html</a>"
},
"author": {
"type": "card",
"name": "Paul Graham",
"url": "https://twitter.com/paulg",
"photo": "https://pbs.twimg.com/profile_images/1824002576/pg-railsconf.jpg"
},
"post-type": "note"
}
},
"_id": "18739751",
"_source": "2773"
}
