Time and again I see the same questions asked: "Why should I use dbt?" or "I don't understand what value dbt offers". So I thought I'd put together an article that touches on some of the benefits, as well as putting together a step through on setting up a new project (using DuckDB as the database), complete with associated GitHub repo for you to take a look at.
Having used dbt since early 2018, and with my partner being a dbt trainer, I hope that this article is useful for some of you. The link is paywall bypassed.
Leveraging Kafka, Polars, and Delta Lake for real-time analytics
If you're a Data Engineer, before long you'll be asked to build a real-time pipeline.
In my latest article, I build a real-time pipeline using Kafka, Polars and Delta tables to demonstrate how these can work together. Everything is available to try yourself in the associated GitHub repo. So if you're curious, take a moment to check out this technical post.
Great point. We use this for our solution design docs, and to display the final star schema in our dbt models that we then embed within our dbt docs. Given we use dbt for our warehouse, we don’t need to worry about the create table statements.
How often do you build and edit Entity Relationship Diagrams? If the answer is ‘more often than I’d like’, and you’re fed up with tweaking your diagrams, take <5 minutes to read my latest article on building your diagrams with code. Track their changes in GitHub, have them build as part of your CI/CD pipeline, and even drop them into your dbt docs if you like.
This is a ‘friends and family’ link, so it’ll bypass the usual Medium paywall.
I’m not affiliated to the tool I’ve chosen in any way. Just like how it works.
Let me know yours thoughts!
The buzz of ‘Big Data’ has passed. Terabytes of data is the new normal, and efficiently managing and processing data is more critical than…
I’ve written a series of Medium articles on creating a Data Pipeline from scratch, using Polars and DeltaTables. The first (linked) is an overview with link to the GitHub repository and each of the deeper dive articles. I then go into the next level of detail, walking through each component.
The articles are paywalled (it took time to build and document), but the link provided is the ‘family & friends’ link which bypasses the paywall for the Lemmy community.
I hope some of you may find this helpful.
I’d never heard of it. Then my first encounter was in a company where someone had built a Python abstraction layer over the top to ‘make it easy’ and it just turned into a mess. But when done well, it’s much better!
A demonstration of how Terraform can be used to manage Snowflake infrastructure - GitHub - nydasco/snowflake-terraform-demo: A demonstration of how Terraform can be used to manage Snowflake infrast...
A few years ago, if you'd mentioned Infrastructure-as-Code (IaC) to me, I would've given you a puzzled look. However I'm now on the bandwagon. And to help others understand how it can benefit them, I've pulled together a simple GitHub repo that showcases how Terraform can be used with Snowflake to manage users, roles, warehouses and databases.
The readme hopefully gives anyone who wants to give it a go the ability to step through and see results. I'm sharing this in the hopes that it is useful to some of you.
A simple speech-to-text and text-to-speech AI chatbot that can be run fully offline. - GitHub - nydasco/jen-ai: A simple speech-to-text and text-to-speech AI chatbot that can be run fully offline.
Hi all,
For those wanting a quick repo to use as a basis to get started, I’ve created jen-ai.
There are full instructions in the readme. Once running you can talk to it, and it will respond.
It’s basic, but a place to start.
While I love to jump on the anti-Elon bus, I have to query: the highest accident rates, or highest accident rates as a percentage of vehicles on the road? If you have 10 Tesla cars on the road, and there are 2 MGs on the road, and 2 Telsas and one MG crashes, then what? 20% of Tesla vs. 50% of MG, but also that could be framed as ‘double the number of Teslas crash compared to MGs’ or ‘Tesla has the highest accident rate of any auto brand’.
It’s running on 2x BlackPills, with hotswap sunset orange switches. The two halves are connected by an ultra thin CAT6 patch cable. It’s the first keyboard I’ve built with backlights, so quite happy that it worked as planned.
It took a while to get used to them (like a day or two). But I’ve been using them on keyboards for a couple of years now, and it feels strange to not have them.
True. I’ve not yet added any code to manage the per key RGB. Hopefully they all work as planned too.
They are custom angled risers I got printed. They fit between the switch and the keycap.
Oh that’s an interesting thought. All might not be lost after all! I’ll need to investigate.
Sam Altman has just been hired by Microsoft. LinkedIn
Blackpill is defined within qmk already. You can do a search in the repo for STM32F401 and see a number of keyboards that use it. From this, you can also see which are split keyboards. The M60 Split is a good example which uses a SPLIT_HAND_PIN to define left and right. The Phoenix is another. This also uses the SPLIT_HAND_PIN. Given that, I've tried updating, but no luck. If SPLIT_KEYBOARD = yes then nothing works. If SPLIT_KEYBOARD = no then they work but they both come through as the left side (even if SPLIT_HAND_PIN B9 is set, and B9 is connected to GND or not).
Hi all,
I'm hoping for some troubleshooting tips. I have a self-build split keyboard using Blackpills.
If I flash either side with 'SPLIT_KEYBOARD = no' in my rules.mk, then both sides work fine (but both working as the left side). If however I set 'SPLIT_KEYBOARD = yes', then both sides stop working.
It's driving me insane!
These are the relevant details of rules.mk:
MCU = STM32F401 BOOTLOADER = stm32-dfu SPLIT_KEYBOARD = yes SERIAL_DRIVER = usart
And config.h:
#pragma once #define HAL_USE_SERIAL TRUE #define SOFT_SERIAL_PIN A12 #define MASTER_LEFT #define MATRIX_ROWS 6 #define MATRIX_COLS 6 #define SERIAL_USART_DRIVER SD1 #define SERIAL_USART_TX_PAL_MODE 7
And halconf.h:
#pragma once #define SERIAL_USB_BUFFERS_SIZE 256 #include_next
And mcuconf.h:
#pragma once #include_next #undef STM32_SERIAL_USE_USART1 #define STM32_SERIAL_USE_USART1 TRUE
Please help! I've been at this for three days.
Are you referring to the #define MASTER_LEFT?
This is just defining whether the USB is plugged into the left board or right board.
All defines start with a hash as standard. It’s not commenting the line out. For that, you’d use //.
Hi all,
I'm hoping for some troubleshooting tips. I have a self-build split keyboard using Blackpills.
If I flash either side with 'SPLIT_KEYBOARD = no' in my rules.mk, then both sides work fine (but both working as the left side). If however I set 'SPLIT_KEYBOARD = yes', then both sides stop working.
It's driving me insane!
These are the relevant details of rules.mk:
MCU = STM32F401 BOOTLOADER = stm32-dfu SPLIT_KEYBOARD = yes SERIAL_DRIVER = usart
And config.h:
#pragma once #define HAL_USE_SERIAL TRUE #define SOFT_SERIAL_PIN A12 #define MASTER_LEFT #define MATRIX_ROWS 6 #define MATRIX_COLS 6 #define SERIAL_USART_DRIVER SD1 #define SERIAL_USART_TX_PAL_MODE 7
And halconf.h:
#pragma once #define SERIAL_USB_BUFFERS_SIZE 256 #include_next
And mcuconf.h:
#pragma once #include_next #undef STM32_SERIAL_USE_USART1 #define STM32_SERIAL_USE_USART1 TRUE
Please help! I've been at this for two days.
Yes. But if 90% of your friends use it, and have groups in it where things are planned and organised, then by not having it you’re going to be missing out on a big chunk of things going on around you.
The challenge is that these days a phone is rarely used for calls or texts, but used with apps like WhatsApp or Teams or Slack or your mobile banking app, or things like that. And so there would need to be a critical mass of these apps to get me to switch.
I’m assuming you downloaded an stl of the switches and the controller? Where did you dl them from?
Thank you. Yeah, the shape is a mix of convex and concave. I might need to just make do for this prototype.
I created a 2d surface that I can perform a linear extrusion on, however the result it obviously a hard edge on the extrusion. I would love to be able to add a bezel - either rounded or at 45 degrees. Is there an easy way?
Thank you! So, you can get a regular 2D shape and use the linear_extrude() function to extrude out at 90 degrees from the object. They refer to in in the docs as the equivalent of pushing playdoh through a press. And from the docs ‘In OpenSCAD Extrusion is always performed on the projection (shadow) of the 2d object xy plane and along the Z axis; so if you rotate or apply other transformations to the 2d object before extrusion, its shadow shape is what is extruded.’
So the Z axis is no related to what I see on the screen, but is in direct relation to the original 2D image. If I were to first rotate the 2D image by 90 degrees, and then extrude, the extrusion would be on the Z axis relative to the 2D object, but the X axis relative to the rest of the project (if that makes sense).
What I want it to extrude on the absolute Z axis of the total project, regardless of the angle of rotation of the 2D object. To put it another way, if I were to have a square on the 2D, and I rotated it 30 degrees before extruding, I would not want a cube to be formed, but rather would want a rhombohedron.
I’ve got the basics down, but now looking to do something more complex, and unsure if it’s possible. I have imported a flat shape (a pcb), and I can extrude this out. What I want to do however is to rotate it on its axis, and then extrude it down the z-axis (not directly out from the surface). Is this possible, and if so, how?
Is there such a thing as an online pcb tester? As in ‘here is my pcb design, when I connect this, I expect it to register as a circuit on these two pins’
Hey there,
Hoping someone could point me to a PCB that is using a Blackpill and a ST7789V OLED display. I have the above and am not 100% on which pins to wire up. GND is obvious. VCC I’m pointing to 3.3V, then there is:
- CS
- DC
- Reset
- SDA
- SCL
The smaller screens only have SDA & SCL. Can anyone confirm whether the others are optional, mandatory, whether there are specific/preferred pins?
I’m feeling a little lost at the moment, so hoping for some guidance.
FYI this is the specific display: here
Hi all,
TL/DR: Ergo Split 36-key with per-key RGB and 2x 2 inch TFT screens, run by 2x BlackPill. Can QMK handle this?
I'm hoping someone can sanity check this for me before I go printing a PCB only to find there is no software capable of doing what I want without major code involvement.
My plan is for a split ergo keyboard with 36 keys (6 rows of 3 per hand, including the thumb cluster). Each key would have an SK6812-MINI-E RGB backlight, and would be run off a BlackPill controller in both sides. I was going to make the connection using a CAT6 patch cable, understanding I don't need all 8 of the cables, but also that a patch cable clicks into place so a reduced chance of shorting anything by disconnecting the sides while still powered.
The gotcha is that I want to run 2x 2" OLED displays, one on each side. Specifically, they are the ST7789V GMT020-02 from Shenzhen GoldenMorning Electronic Co.,Ltd via AliExpress. I know that QMK has Quantum Painter LVGL Integration, and this should support a display. But can I run two independently of one another, one driven off the first BlackPill and the other off the second?
Feedback would be greatly appreciated on this! I have all of the hardware, other than the PCB and case. Before I spend too much time designing the PCB, I want to at least hear that it should be possible, as opposed to hearing that for a variety of reasons it's an absolute non-starter.
Where a comment has been made on my post, it would be awesome to see which post it was related to. It helps give context when there are 15 replies to 5 different posts since I last checked.
Moore's law is the observation that the number of transistors in an integrated circuit (IC) doubles about every two years.
Is there anything similar for the sophistication of AI, or AGI in particular?
I’ve never been much of a musical person. But my kid was playing with a toy piano the other day, pressing buttons and whatnot. I lay my fingers on the keyboard and thought “hmmm… kinda like the home row”.
So my question to the community: have any of you built musical keyboards? Did you post a blog or guide? At first I was thinking a choc switch with a custom long cap that was held off a pivot point at one end and attached to the switch at the other. But musical keyboards vary sound depending on how you press the note. So maybe Hall effect?
Anyway, keen to hear of any adventures down this road by others?
When I eat chicken, I call it chicken. Chicken wing; chicken drumsticks etc.
When I eat lamb, I call it lamb. Lamb shank; lamb cutlets.
So why do I not eat pig or cow? I eat pork or beef. Is there a reason for that?
I currently have an iPhone X. I like it, but it’s gradually dying. It’s also a bit too big. Perfect size would be the iPhone 7 or similar. I’ve had different brand phones before this, so while I’ve been on Apple for a while, I’ve had Samsung, Motorola, Sony, Nokia etc. all the way back to the Nokia 3210.
I’m becoming a bit fed up of big brands and so want to de-Apple myself, but also not go Google. I would love a dumb phone, but I do have some regular apps: MS Auth for work; Lemmy; Philips Hue; Sensibo; WhatsApp.
I was looking at alternative OS options such as Graphene OS, that in theory allow you to run Android apps without Google. But how well do they work?
This would be my primary (only) phone.
So given size considerations (iPhone 7), ability to not be Apple or Google OS, but also be able to run my key apps, what do my options look like?
I’d go into a store and ask, but say ‘not apple and not google’ and you get a vacant stare in response.
Andy @lemmy.world I’m an Australian based Data Engineer, who enjoys making sub-40% custom keyboards.