All this in the final part detective story about the development of Pastilla hardware password Manager open source.
Pastilda hardware password Manager open source. The device allows you to store and enter passwords without the use of software (including command line and BIOS e). The database created using KeePass and is downloaded to the device format .kbdx. The device connects to the computer’s keyboard and press enter “Ctrl ” + ” Shift + ~”. That is why the project was named “Pastilda” (password + tilda “~”).
Learn more about the idea and technical implementation — in article Pastilla — open the hardware password Manager.
After the development and production of a pilot batch, we tried to find customers and buyers in Russia, but further discussion is not reached. Then we decided to raise funds through crowdfunding and stopped on the platform Crowd Supply, which spetsializiruyutsya on open-source hardware projects. We set a goal and eventually exceeded it by almost three times. How does it work? Read the article Pastila: niche crowdfunding
Had to produce and deliver the device to investors.
At the time of launch of the first batch we had applications for 182 Pastille from 149 investors. The orders were from all over the world:
Statistics by country at the moment:
- USA: 143 PCs.
- Germany: 64.
- Australia: 14 PCs
- France: 14 PCs
- Russia: 14 PCs
- Canada: 13 PCs.
- Switzerland: 13 PCs.
- United Kingdom: 13 units
- The Netherlands: 9 units
- Spain: 8 PCs.
- Austria: 5 units
- Denmark: 5 units
- Singapore: 4 PCs.
- China: 3 pieces
- Israel: 3 PCs.
- Belgium: 2 units
- Finland: 2 PCs.
- Ireland: 2 units
- Italy: 2 units
- Japan: 2 PCs.
- Colombia: 1 PC.
- Czech Republic: 1 unit
- Korea: 1 PC.
- Macedonia: 1 PC.
- Malaysia: 1 PC.
- New Zealand: 1 PC.
- Norway: 1 PC.
- Romania: 1 PC.
- Slovenia: 1 PC.
- South Africa: 1 PC.
- Vietnam: 1 PC.
Interestingly, in spite of our articles in Russian, orders from Russia a total of 14 pieces. The production we ordered 324 Pastilla, a number lacked the funds collected.
Due to the specifics of the device we took the advice Crowd Supply and placed production in the U.S. plant Macrofab. A little bit about how it all works. Go to the website and create an order. Downloadable ODB++inside is all about our device: the layers of the Board, a list of components (Bill of materials or BOM), their placement, etc. Then you need to map your BOM with the components available in stock and to confirm the proper placement of the components on the top and bottom of the Board.
Then we get the cost and production time for all parties. The cost is painted in detail: how much labor, how many components, how many of the printed circuit Board, etc.:
Also plotted, which explains the fall in cost during the growth of the party:
The factory undertakes not only for Assembly but also for testing devices. For this we need to write test methods and firmware in a special section. The price firmware about $1 per minute.
Production of a batch of 324 pieces took about 8 weeks. 10% of boards have not been tested with an unclear diagnosis “incorrectly fulfills loader”. Communication was extremely slow, the loss is insignificant, so we just ignored the problem and blamed the 10% in the scrap. Will remember this moment. The running boards were sent to the investors complaints, they did not.
Sales continued, our Pastilla even appeared on Mouser with fierce mark-up on $125. When the Board produced was sold, we decided to run another small batch of 55 devices. This time the production took about 7 weeks, and testing was conducted by the staff of Crowd Supply. Here’s a chart explaining the chronology of orders, shipments and production Pastila:
Manufacture shows the conventional dashes.
Something went wrong
After a while we found out that part of the new batch having problems. 10 Pastila were tested, and the remaining 45 – no. The device shall be stitched as it should, the led is blinking, but the connected keyboard isn’t working. The problem in the production? Curve hands of the tester? Few months, we have tried to solve the problem remotely. Not having succeeded, we asked to send us a few pieces for research. Received the samples and found out that in the scheme of error has crept in. Here:
At the output of the Converter D4:5 TPS76333 voltage of about 4V instead of 3.3 V. Capacity C13 at the output too small. The manufacturer requests a minimum of 4.7 µf, and we have – 1uf. In theory this can lead to the excitation source and increase the voltage at the output. After replacing the tank 10 UF food like it’s back to normal, but the Board was not properly earned.
It’s time to “throw percent”
This means desoldering the microcontroller from the Board and put in a new one or taken from a donor. In this case, we threw MCU Board from the first batch. Worked.
Conclusion: due to the wrong capacitance increases the power of MK, why it gets damaged. Should the STM32 to keep such tension? The documentation says that the maximum voltage without damage to the chip 4B, that is, we went over the edge. But why had this problem not have occured? The colleagues in the US about the findings and asked to verify labeling of controllers from different parties. Labeling different:
What can you learn from marking? According to the article and the information from the description on the microcontroller, we can say the following:
The first batch of China made on the 19th week of 2007 or 2017. Rather 2017, they began to produce a series of F4 in 2011 alone. Audit controller – 2.
The second party of the Philippines, made at 25 weeks of 2017. Audit controller – Y.
Looks like the microcontrollers from the Philippines audit Y less resistant to higher power than the controller of audit 2 from China. We decided to send all devices back on Macrofab, ask them to conduct the research and to change the microcontroller and capacitors on all boards.
The investigation leads Macrofab
There it took the Engineer who used the oscilloscope and the voltmeter, and found our problems. Here’s what he told us:
- 75% of boards are working, that rules out problems with the boards, or errors in the Gerber files
- Positioning(rotation) of the components in order.
- The quality of the soldering, no complaints.
- Measured power supply voltage of 3.3 V and 5V with a multimeter, voltage is normal.
- Checked timing 25MHz.
- Most boards crash during operation. Possible cause – defective microcontroller or fluctuations in supply.
- Dimension power supply 3.3 V oscilloscope showed a voltage of 4.1 V during boot on all boards. It seems the regulator is unable to maintain voltage.
- Maybe some controllers are resistant to increased food, and some don’t.
- The output of the regulator according to Datasheet, should be at least 4.7 UF. Judging by the pattern, capacity, output D4 TPS76333 not sufficient, C13 labeled as 1 UF. Replace the capacitor to 10uf and see how it will affect the stability
- Replacing the capacitor did charge working hours at a time, then it again fails.
- Voltage looks noisier than expected. Recommend to solve this problem before the next production.
The problem was confirmed, most likely we are dealing with errors of design. To production of questions, to help they could not. The Board returned to Crowd supply, and from there sent to us.
The investigation is the Third pin
Finally, we are in the hands of the 41 charges. Well, we are now all correct! The first thing on all boards rewired capacity C13 10 UF. Started testing all boards at a time. Received beautiful cards:
- starts but hangs after a while, the keyboard does not determine
- won’t start at all, no signs of life
- when enabled, starts fast blinking
- included, stitched, hangs the microcontroller heated
Working circuit boards do not. Measured the 3.3 V power from MK, and there is 4 volts! Suddenly aware that the replacement condenser from the source to the high voltage problem does not solve. Watching the oscilloscope, examine the current, changing the source for the LM1117-3.3,
remove the filters and the ground, but nothing has changed – food is still jumps up to 4V.
After enabling some time the voltage is normal, no noise, no buildup of tension, and then abruptly changed to 4B, and lasts so long.
An anomalous behavior of the supply when switching on: power is modulated with pulses of ~100 MS. There was a hypothesis: the power supply is nothing to do with bus power somehow spoils the microcontroller. Maybe with 5V USB port “flows” into the microcontroller through the leg VBUS. In the latest Pastille the leg is not initialized. Tried to forcibly initialize a leg up on the entrance:
We are on the right track! Food is normal, though not immediately – because we have first bootloader works. Go there.
Used loader OpenBLT default initialisere USART1, which intersects with the leg VBUS. And this is Tx, that is, the leg is the output. Output that comes directly from the outside power from a USB port. Looked in the boot, removed the initialization of USART1 – eating has become normal. It is a victory!
The STM32 microcontrollers with supply 3.3 V part of the legs resistant to the 5V TTL signals, in the description they are denoted by FT (Five volt Tolerant). To refresh your memory about these FT legs:
- This property refers to the leg unless it is configured as an input.
- The maximum voltage on the leg 5.5 V.
- The tension on the foot may not exceed the lowest of feeds MK more than 3.6 V.
Here the structural scheme “legs”:
Why leg configured to output, cannot be FT?
It is seen that Output buffer contains two field-effect transistor, the upper shoulder — P-channel and lower N-channel. Although in the block diagram above is shown without a counter-effect transistors diodes, you can be sure that they are there. On the transistors are as follows:
If the P-channel transistor the voltage at the Drain(Drain) exceeds the voltage at the Source(Source) through this diode current will flow until voltages are equal.
Our case is powered via the MK USB LDO is considered in AN4899 separately. Leg VBus 5V power should come only when the microcontroller is already powered. Do this:
What eventually became the cause of the problems in production?
- The program of the microcontroller set up the legs against themselves.
- Because of this increased power and the controller damaged.
- Hardware errors not show.
Now we know how much you can ogresti error in the Board with three connectors and a microcontroller (any number).
We have three of them: two hardware and one software. To understand the true causes of production problems, it took us a year.
Let’s see how many we earned on this project.
Income via Crowd Supply — $14610. Here were the costs:
A large part of the expected is the production of printed circuit boards. Everything else: the Commission Playground, the fee for money transfers, shipping and Assembly of goods in the amount of 24%. The site took 10% Commission from all payments. All the expenses, we instructed the guys from the site, so the money never came to our account.
In the last article we uncovered the complexity of the development around 1840 hours. Imagine that this project was done at commercial rates. Then the development costs can be estimated at $100,000.
To recoup development we need to sell about 3000 Pastilla.
Projects such as Pastilla are born in the hearts of developers and implemented in defiance of common sense.
This approach works with hobby projects, where the main goal is to try new things. But when we are approached with commercial projects, the first thing we do is analyze the economic efficiency. If the economy does not converge, we advise against the launch of the project, starting from the stage of development of TK. We even have the idea for a new article. Something like “10 reasons why you need to develop your device”. Actual topic?
The main conclusion that we made during the work on this project: in this version Pastilla is only suitable for personal use. If we want the device solved the problem on the business level – one hardware is not enough to build a system.
We have the following ideas:
- To combine a physical and a server (or cloud) storage password. Conventionally, split the password into two parts. The first part would be stored on the physical key (as now implemented), the second at the time of input of a password is requested from the server and transmitted to the key, such as Wi-Fi.
- Similar to the first embodiment, but keep in the key is not part of the password and private encryption key that will decrypt the password coming from the server via Wi-Fi (password is pre-encrypted with another key).
We welcome feedback from colleagues in the field of information security.
You must upgrade the loader Pastille to prevent damage to the MK. Updated the boot loader lies in the repository. There is instructions on how to flash Pastilu. Solder the capacitor for stable operation is possible, but not necessarily.
Again we apologize for the delay in buyers who are still waiting for your Pastille. All of the remaining 40 Pasting will fly to you in the coming weeks.
Initialize the pins wisely, take care of exits from the youth. And of course, use strong passwords.