The key to getting people to at least appreciate logical data models is to just do them as part of whatever modeling effort you are working on. Don’t say “stop”. Just model on. Demonstrate, don’t tell your teams where the business requirements are written down, where they live. Then demonstrate how that leads to beautiful physical models as well.
Tag Archives: tech blog
Tesla Battery Economics: On the Path to Disruption
At the utility scale, it may actually be even more disruptive. Tesla appears to be selling the utility scale models at < $250 / kwh. Multiple utility studies suggest that such a price should replace natural gas peakers and drive gigantic grid-level deployments.
Source: Tesla Battery Economics: On the Path to Disruption | Ramez Naam
Tentative Conclusion: The battery is right on the verge of being cost effective to buy across most of the US for day/night arbitrage. And it’s even more valuable if outages come at a high economic cost.
Decertifying the worst voting machine in the US
I’ve been in the security field for 30 years, and it takes a lot to surprise me. But the VITA report really shocked me – as bad as I thought the problems were likely to be, VITA’s five-page report showed that they were far worse. And the WinVote system was so fragile that it hardly took any effort. While the report does not state how much effort went into the investigation, my estimation based on the description is that it was less than a person week.
via Decertifying the worst voting machine in the US.
So how would someone use these vulnerabilities to change an election?
- Take your laptop to a polling place, and sit outside in the parking lot.
- Use a free sniffer to capture the traffic, and use that to figure out the WEP password (which VITA did for us).
- Connect to the voting machine over WiFi.
- If asked for a password, the administrator password is “admin” (VITA provided that).
- Download the Microsoft Access database using Windows Explorer.
- Use a free tool to extract the hardwired key (“shoup”), which VITA also did for us.
- Use Microsoft Access to add, delete, or change any of the votes in the database.
- Upload the modified copy of the Microsoft Access database back to the voting machine.
- Wait for the election results to be published.
The freedom to tinker blog has been doing research on voting machines for a very long time although in this case they are reporting the results of research done by Virginia IT people in their decertification. In the past most vulnerabilities uncovered required physical access to a voting machine and a bit of skullduggery making it difficult to change votes on a large scale. I simply cannot comprehend for what purpose these voting devices needed to be on a wifi network other than someone thought it was “cool.” This entire report is mind boggling and makes me wonder how many more areas of the country are doing this now.
Surviving Data Science “at the Speed of Hype”
A good predictive model requires a stable set of inputs with a predictable range of values that won’t drift away from the training set. And the response variable needs to remain of organizational interest.
via Surviving Data Science “at the Speed of Hype” – John Foreman, Data Scientist.
If you want to move at the speed of “now, light, big data, thought, stuff,” pick your big data analytics battles. If your business is currently too chaotic to support a complex model, don’t build one. Focus on providing solid, simple analysis until an opportunity arises that is revenue-important enough and stable enough to merit the type of investment a full-fledged data science modeling effort requires.
Clues In Sony Hack Point To Insiders
Researchers from the security firm Norse allege that their investigation of the hack of Sony has uncovered evidence that leads, decisively, away from North Korea as the source of the attack. Instead, the company alleges that a group of six individuals is behind the hack, at least one a former Sony Pictures Entertainment employee who worked in a technical role and had extensive knowledge of the company’s network and operations.
via A New Script: Clues In Sony Hack Point To Insiders | The Security Ledger.
Things You Should Never Do, Part I – Joel on Software
There’s a subtle reason that programmers always want to throw away the code and start over. The reason is that they think the old code is a mess. And here is the interesting observation: they are probably wrong. The reason that they think the old code is a mess is because of a cardinal, fundamental law of programming:
It’s harder to read code than to write it.
via Things You Should Never Do, Part I – Joel on Software.
Each of these bugs took weeks of real-world usage before they were found. The programmer might have spent a couple of days reproducing the bug in the lab and fixing it. If it’s like a lot of bugs, the fix might be one line of code, or it might even be a couple of characters, but a lot of work and time went into those two characters.
When you throw away code and start from scratch, you are throwing away all that knowledge. All those collected bug fixes. Years of programming work.
Remove duplicates without sorting file
Usually whenever we have to remove duplicate entries from a file, we do a sort of the entries and then eliminate the duplicates using “uniq” command.
But if we have to remove the duplicates and preserve the same order of occurrence of the entries, here is the way:
via UNIX Command Line: Remove duplicates without sorting file – BASH.
$ awk ‘ !x[$0]++’ file3
From: Unix: removing duplicate lines without sorting
This command is simply telling awk which lines to print. The variable $0 holds the entire contents of a line and square brackets are array access. So, for each line of the file, the node of the array named x is incremented and the line printed if the content of that node was not (!) previously set.
Linux Command: xxd
[rmiller@pacific]# echo "hello world" > hello
[rmiller@pacific]# xxd hello
0000000: 6865 6c6c 6f20 776f 726c 640a hello worldSo you can use this tool to byte edit files. One rather unusual use I’ve found for it is to paste in an RPM to a system that I only had serial console access to. I just ran xxd on it, copied it into the buffer, and pasted it into a file on the remote server. A quick xxd -r, and voila. RPM.
I recently ran across the above blog entry which is from 2010. All these years working with Unix systems and I never knew about this command. When I parse a web site to extract information it’s necessary to output clean and concise ASCII data for my downstream scripts. My perl scripts that html parse do filter this out but sometimes a funny character gets through. Normally I have been using hexedit to determine the hex code of the offending character and although it works, it’s not as elegant as the above xxd command. Now I can do the following:
xxd offendingdatafile.txt | grep "mystring" | more
The above should simply output lines containing the offending hexcode using grep if I kind of notice a unique searchable string (mystring) before the offending hex character. I could also:
xxd offendingdatafile.txt > myfile.dat
vi myfile.dat
Instead of using clunky hexedit to search for mystring I can use good old vi.
I’m sure there are lots of other uses for this utility — especially in shell scripts. Unix has so many commands and I utilize a subset adequate to getting whatever it is I need to do. Every year I pick up one or two new useful commands that are more efficient and xxd is one of them.
Why the Z-80’s data pins are scrambled
I have been reverse-engineering the Z-80 processor using images and data from the Visual 6502 team. The image below is a photograph of the Z-80 die. Around the outside of the chip are the pads that connect to the external pins. (The die photo is rotated 180° compared to the datasheet pinout, if you try to match up the pins.) At the right are the 8 data pins for the Z-80’s 8-bit data bus in a strange order.
via Ken Shirriff’s blog: Why the Z-80’s data pins are scrambled.
The motivation behind splitting the data bus is to allow the chip to perform activities in parallel. For instance an instruction can be read from the data pins into the instruction logic at the same time that data is being copied between the ALU and registers. The partitioned data bus is described briefly in the Z-80 oral history[3], but doesn’t appear in architecture diagrams.
The complex structure of the data buses is closely connected to the ordering of the data pins.
Create an Army of Raspberry Pi Honeypots on a Budget
Organizations typically focus on monitoring inbound and outbound network traffic via firewalls, yet ignore internal network traffic due to the complexity involved. In the scenario above, a firewall will not protect or alert us.
By running honeypots on our internal network, we are able to detect anomalous events. We gain awareness and insight into our network when network hosts interact with a Raspberry Pi honeypot sensor. Since there isn’t a good reason to interact with it (since it doesn’t do anything), activity on the Raspberry Pi is usually indicative of something roaming around our network and a possible security breach.
via Create an Army of Raspberry Pi Honeypots on a Budget | ThreatStream.