New – Lifecycle Management for Amazon EBS Snapshots

It is always interesting to zoom in on the history of a single AWS service or feature and watch how it has evolved over time in response to customer feedback. For example, Amazon Elastic Block Store (EBS) launched a decade ago and has been gaining more features and functionality every since. Here are a few of the most significant announcements:

  • August 2008 – We launched EBS in production form, with support for volumes of up to 1 TB and snapshots to S3.
  • September 2010 – We gave you the ability to Tag EBS Volumes.
  • August 2012 – We launched Provisioned IOPS for EBS volumes, allowing you to dial in the level of performance that you need.
  • June 2014 – We gave you the ability to create SSD-backed EBS volumes.
  • March 2015 – We gave you the ability to create EBS volumes of up to 16 TB and 20,000 IOPS.
  • April 2016 – We gave you New cold storage and throughput options.
  • June 2016 – We gave you the power to create Cross-account copies of encrypted EBS snapshots.
  • February 2017 – We launched Elastic Volumes, allowing you to adjust the size, performance, and volume type of an active, mounted EBS volume.
  • December 2017 – We gave you the ability to create SSD-backed volumes that deliver up to 32,000 IOPS.
  • May 2017 – We launched Cost allocation for EBS snapshots so that you can assign costs to projects, departments, or other entities.
  • April 2018 – We gave you the ability to Tag EBS snapshots on creation and to Use resource-level permissions to implement stronger security policies.
  • May 2018 – We announced that encrypted EBS snapshots are now stored incrementally, resulting in a performance improvement and cost savings.

Several of the items that I chose to highlight above make EBS snapshots more useful and more flexible. As you may already know, it is easy to create snapshots. Each snapshot is a point-in-time copy of the blocks that have changed since the previous snapshot, with automatic management to ensure that only the data unique to a snapshot is removed when it is deleted. This incremental model reduces your costs and minimizes the time needed to create a snapshot.

Because snapshots are so easy to create and use, our customers create a lot of them, and make great use of tags to categorize, organize, and manage them. Going back to my list, you can see that we have added multiple tagging features over the years.

Lifecycle Management – The Amazon Data Lifecycle Manager
We want to make it even easier for you to create, use, and benefit from EBS snapshots! Today we are launching Amazon Data Lifecycle Manager to automate the creation, retention, and deletion of Amazon EBS volume snapshots. Instead of creating snapshots manually and deleting them in the same way (or building a tool to do it for you), you simply create a policy, indicating (via tags) which volumes are to be snapshotted, set a retention model, fill in a few other details, and let Data Lifecycle Manager do the rest. Data Lifecycle Manager is powered by tags, so you should start by setting up a clear and comprehensive tagging model for your organization (refer to the links above to learn more).

It turns out that many of our customers have invested in tools to automate the creation of snapshots, but have skimped on the retention and deletion. Sooner or later they receive a surprisingly large AWS bill and find that their scripts are not working as expected. The Data Lifecycle Manager should help them to save money and to be able to rest assured that their snapshots are being managed as expected.

Creating and Using a Lifecycle Policy
Data Lifecycle Manager uses lifecycle policies to figure out when to run, which volumes to snapshot, and how long to keep the snapshots around. You can create the policies in the AWS Management Console, from the AWS Command Line Interface (CLI) or via the Data Lifecycle Manager APIs; I’ll use the Console today. Here are my EBS volumes, all suitably tagged with a department:

I access the Lifecycle Manager from the Elastic Block Store section of the menu:

Then I click Create Snapshot Lifecycle Policy to proceed:

Then I create my first policy:

I use tags to specify the volumes that the policy applies to. If I specify multiple tags, then the policy applies to volumes that have any of the tags:

I can create snapshots at 12 or 24 hour intervals, and I can specify the desired snapshot time. Snapshot creation will start no more than an hour after this time, with completion based on the size of the volume and the degree of change since the last snapshot.

I can use the built-in default IAM role or I can create one of my own. If I use my own role, I need to enable the EC2 snapshot operations and all of the DLM (Data Lifecycle Manager) operations; read the docs to learn more.

Newly created snapshots will be tagged with the aws:dlm:lifecycle-policy-id and  aws:dlm:lifecycle-schedule-name automatically; I can also specify up to 50 additional key/value pairs for each policy:

I can see all of my policies at a glance:

I took a short break and came back to find that the first set of snapshots had been created, as expected (I configured the console to show the two tags created on the snapshots):

Things to Know
Here are a couple of things to keep in mind when you start to use Data Lifecycle Manager to automate your snapshot management:

Data Consistency – Snapshots will contain the data from all completed I/O operations, also known as crash consistent.

Pricing – You can create and use Data Lifecyle Manager policies at no charge; you pay the usual storage charges for the EBS snapshots that it creates.

Availability – Data Lifecycle Manager is available in the US East (N. Virginia), US West (Oregon), and EU (Ireland) Regions.

Tags and Policies – If a volume has more than one tag and the tags match multiple policies, each policy will create a separate snapshot and both policies will govern the retention. No two policies can specify the same key/value pair for a tag.

Programmatic Access – You can create and manage policies programmatically! Take a look at the CreateLifecyclePolicy, GetLifecyclePolicies, and UpdateLifeCyclePolicy functions to get started. You can also write an AWS Lambda function in response to the createSnapshot event.

Error Handling – Data Lifecycle Manager generates a “DLM Policy State Change” event if a policy enters the error state.

In the Works – As you might have guessed from the name, we plan to add support for additional AWS data sources over time. We also plan to support policies that will let you do weekly and monthly snapshots, and also expect to give you additional scheduling flexibility.

— Jeff;

Elon Musk’s Flint Water Plan Misses the Point

Wednesday afternoon, on the heels of his belated effort to rescue a youth soccer team from a Thai cave with a tiny submarine, Elon Musk promised to fix another seemingly intractable problem. “Please consider this a commitment that I will fund fixing the water in any house in Flint that has water contamination above FDA levels,” Musk wrote in a tweet. “No kidding.”

You can nitpick pieces of this—the EPA, not the FDA, determines how many parts per billion of lead is safe in drinking water—or dismiss it as just another manifestation of Musk’s itinerant savior complex. But know that Flint, at least, welcomes Musk’s help. Just maybe not the version that’s on offer.

Which, in fairness, continues to evolve. Musk went on to invite residents to tweet their water quality test results to him—no takers yet, it seems—and said he would send someone over to install a water filter. When a reporter suggested that many Flint houses have safe water already, Musk pivoted to organizing “a weekend in Flint to add filters” to the remaining houses that lack them.

‘There are many people in Flint, I think it’s safe to say, who are never going to trust tap water again.’

Benjamin Pauli, Kettering University

Flint does need help, but filters are one thing it already has plenty of; the city distributes those and water testing kits, for free, at City Hall, and will continue to until Flint’s remaining 14,000 damaged lead and galvanized water service pipes have been fully replaced. And even then, slapping a filter on a kitchen faucet doesn’t address the deep-seated problems still felt by the Flint community four years after its crisis began.

“We had a lot of things damaged as a result of the corrosive water,” says Flint Mayor Karen Weaver, who offered in a tweet Wednesday to talk through her city’s “specific needs” with Musk. “This is about reestablishing trust, and rebuilding trust. While filters have been helpful, we still need access to bottled water. People need to see all new pipes going in. That’s how you’re going to reestablish trust. And we know that’s what the residents deserve.”

Musk took her up on it, suggesting he’d call on Friday. Weaver says her office and Musk’s are still sorting out schedules, but preliminary conversations have been promising.

Filtering Down

It’s worth spending more time talking about those filters, not because they demonstrate Musk’s lack of familiarity with Flint’s current situation, but because they underscore the city’s deeper challenges.

First, it’s important to note that Flint’s drinking water has met federal standards for contaminants for at least a year. “From every objective measure that is out there, Flint’s water is like any other US city with old lead pipes,” says Siddhartha Roy, who works on the Virginia Tech research team that helped shed light on the Flint water crisis and has tracked it ever since. Water from old lead pipes still isn’t ideal, obviously, and makes filters a necessity. But even then, Flint residents remain understandably wary.

“There are many people in Flint, I think it’s safe to say, who are never going to trust tap water again under any circumstances,” says Benjamin Pauli, a social scientist at Flint’s Kettering University, who has been involved in clean water activism efforts. “It’s true that the filters solve a lead problem at point of use, but there are lots of other issues with the filters.”

Not all residents know how to install and maintain them, for one. A March survey of 2,000 residents by Flint News showed that 15 percent of respondents didn’t have a filter, while over a third weren’t confident in their ability to change the filter at the appropriate time.

And then there’s what Roy calls the “big trust gap” that makes Flint activists and residents suspicious of even working filters. That’s because they effectively get lead out of the water at a specific tap, but don’t clear away bacteria. For a city that suffered a deadly spike in Legionnaires’ disease in 2016, which has been linked to corrosive water from the Flint River, that causes understandable unease. But Roy notes that the current bacteria found in Flint’s filters has not been shown to be harmful. And anyone who does have concerns can follow a few simple steps to minimize bacterial buildup.

“We do have concerns about filter use, and maintenance, and education around the filters. Everybody is not comfortable with that. Seniors are especially not comfortable with the filters,” says Weaver, who notes that the city does have Community Outreach and Resident Education that visits homes to help remediate any filter issues that arise.

Which again should sound familiar to anyone who read Musk’s tweets. What he proposes to accomplish in a barnstorming weekend has been an available resource for years. Better, then, to focus on what Flint really needs.

Bottle It Up

In April, the state of Michigan stopped providing free bottled water to Flint. For a city that still doesn’t trust its taps, the impact can’t be overstated.

“The bottled water is necessary as a short-term intervention for a long-term, structural water system problem,” says Pastor Monica Villarreal, who has helped organize community-based efforts to provide clean water resources in Flint. “The water crisis is going to affect this city from generation to generation. And when you look at it from that perspective, two, three, maybe even four years of bottled water is not much.”

Community aid stations that were once open daily to distribute bottled water now operate just three times a week. And in the absence of state support, Flint increasingly has to rely on private donors; Weaver says the Detroit Police Department recently brought in a fresh supply.

So if Elon Musk—or anyone else—wants to help Flint, start with bottled water, which residents will continue to depend on until every last lead and galvanized line gets replaced. “Bottled water is really the life and death issue,” Villarreal says.

‘That was one of the fears of the residents, that attention would go away, and we have not been made whole.’

Flint Mayor Karen Weaver

And if you want to think bigger, plenty of options remain. “One issue that residents have been raising from very early on is that corrosive water from the river didn’t just damage service lines and water mains, it also damaged the plumbing within people’s homes,” says Kettering’s Pauli. “And not just pipes but fixtures, and also appliances that use water. That would include washing machines, and dishwashers, and hot water heaters.”

Scale it up again, to billionaire proportions. “We want to look at the bigger infrastructure issues in the city as well,” Weaver says. “It’s about reestablishing trust. You have to be confident in the water again.” One way to accomplish that? Get more contractors on the ground replacing service lines; get a three-year replacement plan finished by the end of 2018. And then, Weaver says, look at investment in the community. Instead of—or in addition to—giving people water, how can you help get them back to work?

Those are the types of questions Elon Musk can expect on his call with the mayor. But no matter what comes of it, even expressing interest in the first place has accomplished something invaluable: Reminding people that Flint still exists, and still needs help.

“We’re glad to have the attention. That was one of the fears of the residents, that attention would go away, and we have not been made whole,” Weaver says. “We want everybody watching, because what happened to Flint should never happen to any place again.”


More Great WIRED Stories

It Just Got Easier for the FCC to Ignore Your Complaints

It may soon be harder to get the Federal Communications Commission to listen to your complaints about billing, privacy, or other issues with telecommunications carriers like AT&T and Verizon.

Today, the agency approved changes to its complaint system that critics say will undermine the agency’s ability to review and act on the complaints it receives.

On Wednesday, The Washington Post reported that the controversial changes had been dropped from the proposal, but the commission voted 3–1 along party lines to approve it with the changes intact.

“I believe we should be doing everything within our power to make it easier for consumers to file complaints and seek redress,” Jessica Rosenworcel, the FCC’s lone Democratic commissioner, said during today’s meeting. “This decision utterly fails that test.”

The FCC has two complaint systems. Formal complaints cost $225 to file and work a bit like a court proceeding. The informal complaint system is free. According to the FCC website, the agency doesn’t work to resolve individual informal complaints, but reviews them for trends or patterns that can lead to investigations or actions against carriers.

The changes approved today mostly deal with formal complaints about utility poles. But they include small changes to the informal complaint system that critics say will have an outsized impact on how the agency handles complaints.

At issue is the removal of the words “review and disposition” from the informal complaint rules. The term “disposition” means “resolution.”

In a letter on Tuesday, two Democrats in the House of Representatives argued that under the revised rule, FCC staffers would forward consumer complaints to the targeted company, and advise to file a formal complaint, for $225, if they’re not satisfied with the company’s response.

An FCC spokesman told WIRED Wednesday the change to the informal complaint process was only intended to clarify that the agency doesn’t act on individual complaints.

But critics worry that by removing the reference to review and disposition, FCC staff will no longer have the authority to review and act on informal complaints.

“Now the FCC can ignore informal complaints completely if it wants to,” says Gigi Sohn, a former FCC lawyer who is now a fellow at the Georgetown Law Institute for Technology Law and Policy. “This FCC’s contempt for the public it is legally mandated to serve is remarkable.”


More Great WIRED Stories

Judd Legum’s ‘Popular Information’ Is a Politics Newsletter for Everyone

One of the few things people agree on in 2018 is that the news industry is broken. The old business models don’t work. Meanwhile, audiences feel overwhelmed and underserved: According to a recent Pew Research Center survey, seven in 10 Americans say they are exhausted by the news. The consensus stops with the diagnosis, though; when it comes to prescribing a treatment, everyone has different ideas.

To Judd Legum, editor in chief and founder of left-leaning political news website ThinkProgress, the two biggest problems are ads and social media. Digital ads aren’t sustainable as a business model for online publications and they create incentives for clickbait and other poor-quality journalism. Social media is a firehouse of information and leave readers and outlets alike at the whim of algorithms. This is especially worrisome to Legum right now, given the upcoming midterm elections and the need for voters to be informed on the issues.

“People need to make more intentional choices and to regain power over what news they read,” says Legum. “There’s something fundamentally broken about news delivery as a process. The power is too concentrated. I’ve felt more and more strongly that I wanted to start something new that could circumvent the system.”

Today, Legum is joining a small but growing group of journalists and readers who think one way to fix this is through a good old-fashioned email newsletter. And he is going all in. After 13 years at the helm of ThinkProgress–a site that garners around 10 million unique visitors a month–he’s leaving the 40-person newsroom he runs to launch a paid political newsletter called “Popular Information”, which he will write himself. Starting July 23, Legum will publish “Popular Information” four days a week. He says it will be a mix of deep reporting and analysis, focused on national issues with a progressive lens.

The benefits to both journalist and reader of a direct-to-inbox newsletter are clear: there’s no middleman between reader and writer, no algorithm deciding what you see and what you don’t. And it’s a relationship built on trust—something that the media needs to rebuild with Americans after years of declining public opinion. Readers explicitly opt in to receive newsletters, with the expectation that they will deliver something of value. “It’s intimate to come into somebody’s inbox every day. Email is a more intimate medium than just publishing on the web,” says Jay Rosen, professor of journalism at NYU’s Arthur L Carter Journalism Institute.

‘People need to make more intentional choices and to regain power over what news they read.’

Judd Legum

That’s part of what is so appealing to Legum, who came up as a blogger in the early aughts, when loyal readers visited and often commented on their favorite blogs every day. Once social media rendered that behavior obsolete, Twitter became the place for writers and readers to have a direct relationship, but that introduces a host of new problems.

“Twitter is very ephemeral,” Legum says, adding that most of what he tweets is in reaction to something immediate. “What I’m trying to do with the newsletter is provide some perspective and organization for people who might have a real job during the day. This is for people who are feeling overwhelmed.”

And he’s hoping a good number of his readers will pay for that curation. “Popular Information” will be free for everyone for the first six to eight weeks in order to gain an audience; after that, the Monday edition will be free, and the other three days accessible only to paying members. Luckily, the overhead will be low. Legum will work out of his small apartment in Washington DC and has enough money saved to live off for a little while he builds up his subscriber base.

“There’s a hustle to it,” says Legum. If he succeeds, he might expand “Popular Information” to have a staff larger than one. Even if he does there are downsides to the paid model: cost of entry makes information inaccessible to some.

“All the idealism of journalism is that you can equip the public with information so that it knows what’s going on in its world. So there is an element of all subscription products that is in a sense anti-public,” Rosen says.

It’s a tension that besets any paywall, and it’s something Legum has considered. The name of his newsletter comes from a line James Madison wrote in a letter in 1822: “A popular Government, without popular information, or the means of acquiring it, is but a Prologue to a Farce or a Tragedy; or, perhaps both.” To make “Popular Information” as accessible as possible, Legum plans to keep the subscription cost low. Though he hasn’t decided exactly how much yet, it’ll be less than $10 a month.

Newsletters have long been a way for media outlets to directly reach their audiences, for free or for a price. So why doesn’t Legum just launch “Popular Information” as part of ThinkProgress?

“ThinkProgress is a full-time job. We produce about 25 pieces per day and have three dozen staffers. So, in my view, I don’t have time to do this and my current job. I need to be able to focus my attention on this so I can do it right,” he says. Think Progress managing editor Tara Culp-Ressler will take over his duties until a new EIC can be found. She told WIRED in a statement: “The ThinkProgress team is grateful for the newsroom that Judd built. Obviously we’re sad to see him go, but we’re excited to watch his next chapter unfold.”

Legum’s also ready for something new, and sees a dearth of low-cost, high-quality newsletters focused on politics for a general audience, even as newsletter-first publications have taken off.

In recent years some have gained massive audiences, like Gwyneth Paltrow’s Goop, which has morphed into a lifestyle brand, or The Skimm, which aggregates news from across the web and this year raised $12 million from the likes of Google. The model Legum plans to follow most closely comes from tech analyst Ben Thompson, whose daily newsletter Stratechery costs $10 a month or $100 a year, and is required reading for many people interested in tech.

But the biggest political newsletters right now come from news organizations like Axios and Politico. Legum notes that Axios’ morning and evening newsletters are sponsored by Wall Street–Goldman Sachs, Bank of America. Politico’s Pro subscription, which includes much more than newsletters, is so expensive that even with only 20,000 subscribers it accounted for half the company’s revenue in 2017; at the time, a five-person subscription started at $8,000 a year. Its free newsletter, Playbook, grows out of that insider perspective, in Legum’s opinion, treating politics like a spectator sport for elites rather than something that affects people’s lives. He hopes by offering something without corporate money, paid instead in small amounts by individual stakeholders who want to read what he has to say, that “Popular Information” will act as a guide to politics that matter.

The other thing Legum’s counting on to pull this off is a streamlined back-end. Whereas Thompson, who launched Stratechery in 2014, had to cobble together the means to produce his newsletter himself, Legum has Substack, a startup founded last year by Hamish McKenzie, a former journalist, and Chris Best and Jairaj Sethi, both formerly of messaging app Kik. Early on, they consulted with Thompson and other newsletter producers and recall hearing over and over that half of their time was spent renewing subscriptions and managing the newsletter.

Substack deals with all of that, taking payments, distributing the newsletter to people’s inboxes, renewing subscriptions, and making sure everything works technically. In exchange, it takes a 10 percent cut for newsletters that charge subscribers (for everyone else, publishing is free while the service is in beta).

“We were thinking about how bad incentives for online advertising have sort of broken the media,” says Best, Substack’s CEO. “They incentivize clickbait and cheap outrage in a way that’s dissatisfying for everybody. We’re caught in this bad equilibrium where everybody has to write clickbait stuff.”

Substack graduated out of YCombinator last winter and has raised $2 million in funding. Earlier this week, Best and McKenzie told Nieman Lab that across its hundreds of existing newsletters it has hit 11,000 paid subscribers, who are paying an average of just under $80 a year. And approximately 40 newsletter creators are making what Best and McKenzie told Nieman Lab was “meaningful money”–though “meaningful” can mean different things to different people.

“I don’t really have any expectations on money except I’m going to put my full effort into this and see what I can make of it,” Legum says. “Whether I succeed or not I think depends on whether it ends up being good.”

One challenge facing Legum, and any other newsletter creator, is that at some point people will hit a limit on how many newsletters they want to receive and are willing to pay for. “Popular Information” will be competing for your money with all other paid publications, like newspapers and websites like WIRED, which has a paywall. For now, Legum hopes he’s getting in to the political newsletter game at a time when people are hungry for in-depth information, and interested in receiving it from someone who doesn’t have a corporate sponsor. He also has the benefit of a loyal readership at ThinkProgress who he hopes will sign up, credibility working in and covering politics for 15 years, and 280,000 Twitter followers.

As Rosen notes, the first hurdle to a business model like this is to get anyone to sign up. Having an already established audience certainly helps. So far Substack’s biggest hits are written by well-known writers such as Rolling Stone’s Matt Taibbi and Slate’s Daniel Ortberg. Taibbi has teamed up with an anonymous drug-dealing friend to write a fictional work in newsletter installments, and Ortberg writes a quirky humor newsletter called the “Shatner Chatner”. “Popular Information” will be the first political dispatch for the company.

And though Legum will be a bit busy in the weeks and months to come, he promises to keep tweeting.


More Great WIRED Stories

AWS Storage Gateway Recap – SMB Support, RefreshCache Event, and More

To borrow my own words, the AWS Storage Gateway is a service that includes a multi-protocol storage appliance that fits in between your existing application and the AWS Cloud. Your applications see the gateway as a file system, a local disk volume, or a Virtual Tape Library, depending on how it was configured.

Today I would like to share a few recent updates to the File Gateway configuration of the Storage Gateway, and also show you how they come together to enable some new processing models. First, the most recent updates:

SMB Support – The File Gateway already supports access from clients that speak NFS (versions 3 and 4.1 are supported). Last month we added support for the Server Message Block (SMB) protocol. This allows Windows applications that communicate using v2 or v3 of SMB to store files as objects in S3 through the gateway, enabling hybrid cloud use cases such as backup, content distribution, and processing of machine learning and big data workloads. You can control access to the gateway using your existing on-premises Active Directory (AD) domain or a cloud-based domain hosted in AWS Directory Service, or you can use authenticated guest access. To learn more about this update, read AWS Storage Gateway Adds SMB Support to Store and Access Objects in Amazon S3 Buckets.

Cross-Account Permissions – Some of our customers run their gateways in one AWS account and configure them to upload to an S3 bucket owned by another account. This allows them to track departmental storage and retrieval costs using chargeback and showback models. In order to simplify this important use case, you can configure the gateway to provide the bucket owner with full permissions. This avoids a pain point which could arise if the bucket owner was unable to see the objects. To learn how to set this up, read Using a File Share for Cross-Account Access.

Requester Pays – Bucket owners are responsible for storage costs. Owners pay for data transfer costs by default, but also have the option to have the requester pay. To support this use case, the File Gateway now supports S3’s Requester Pays Buckets. Data collectors and aggregators can use this feature to share data with research organizations such as universities and labs without incurring the costs of access themselves. File Gateway provides file based access to the S3 objects, caches recently accessed data locally, helping requesters reduce latency and costs. To learn more, read about Creating an NFS File Share and Creating an SMB File Share.

File Upload Notification – The gateway caches files locally, and uploads them to a designated S3 bucket in the background. Late last year we gave you the ability to request notification (in the form of a CloudWatch Event) when new files have been uploaded. You can use this to initiate cloud-based processing or to implement advanced logging. To learn more, read Getting File Upload Notification and study the NotifyWhenUploaded function.

Cache Refresh Event – You have long had the ability to use the RefreshCache function to make sure that the gateway is aware of objects that have been added, removed, or replaced in the bucket. The new Storage Gateway Cache Refresh Event lets you know that the cache is now in sync with S3, and can be used as a signal to initiate local processing. To learn more, read Getting Refresh Cache Notification.

Hybrid Processing Using File Gateway
You can use the File Upload Notification and Cache Refresh to automate some of your routine hybrid process tasks!

Let’s say that you run a geographically distributed office or retail business, with locations all over the world. Raw data (metrics, cash register receipts, or time sheets) is collected at each location, and then uploaded to S3 using a File Gateway hosted at each location. As the data arrives, you use the File Upload Notifications to process each S3 object, perhaps using an AWS Lambda function that invokes Amazon Athena to run a stock set of queries against each one. The data arrives over the course of a couple of hours, and results accumulate in another bucket. At the end of the reporting period, the intermediate results are processed, custom reports are generated for each branch location, and then stored in another bucket (this bucket, as it turns out, is also associated with a gateway, and each gateway will have cached copies of the prior versions of the reports). After you generate your reports, you can refresh each of the gateway caches, wait for the corresponding notifications, and then send an email to the branch managers to tell them that their new report is available.

Here’s a video (and presentation) with more information about this processing model:

Now Available
All of the features listed above are available now and you can start using them today in all regions where Storage Gateway is available.

— Jeff;

Teleconsole – Share Your Linux Terminal with Your Friends

Teleconsole is a free open source and powerful command line tool for sharing your Linux terminal session with people you trust. Your friends or team members can connect to your Linux terminal session via a command-line over SSH or via a browser over HTTPS protocol.

How does it work?

Teleconsole is a clustered SSH server with a built-in SSH proxy and was written in GoLang. You can use this tool to launch secure SSH sessions, perform forwarding of local TCP ports and setup private proxies.

After you launch teleconsole on your system, it opens a new shell session and prints the unique session ID as well as a WebUI link which you need to share with your friends, for them to join via a command line over SSH or from their web browsers over HTTPS.

Read Also: GoTTY – Share Your Linux Terminal (TTY) as a Web Application


Additionally, teleconsole also enables for forwarding local TCP ports, thus allowing your friends to access the web applications running on your localhost in case it’s behind NAT.

Warning: Teleconsole comes with certain security risks that you should take note of; it creates an SSH server accessible via public Internet during the Teleconsole session, this will practically give your keyboard to anyone with a link.

How to Install Teleconsole in Linux

The fastest and easiest way to install Teleconsole on your Linux distribution is to type the following command in your terminal.

$ curl https://www.teleconsole.com/get.sh | sh

Once Teleconsole installed, you can start it by typing the following command. This is very useful when you get stuck in some configuration on a Linux box behind NAT. Just invite and share your Linux session with your friend to help you.

$ teleconsole
Sample Output
Starting local SSH server on localhost...
Requesting a disposable SSH proxy on as.teleconsole.com for tecmint...
Checking status of the SSH tunnel...
Your Teleconsole ID: asce38b0cbb9db97ef16562d1feffe5b84c9a204b8
WebUI for this session: https://as.teleconsole.com/s/ce38b0cbb9db97ef16562d1feffe5b84c9a204b8
To stop broadcasting, exit current shell by typing 'exit' or closing the window.

Next, copy the unique session ID printed or WebUI link and share it via a secure means with people you trust. Your friends can join in using the session ID as shown.

$ teleconsole join asce38b0cbb9db97ef16562d1feffe5b84c9a204b8

Or they can join by clicking on a WebUI link to access it via a web browser as shown in the following screen shot.

Share Linux SSH Session via Browser

Share Linux SSH Session via Browser

Now you and your friend both are using the same Linux terminal session running on your system, even if both of you are on different networks separated by NAT.

To stop broadcasting, exit the current shell by typing ‘exit’ command or closing the terminal window.

$ exit

How to Enable Port Forwarding

Another important feature of Teleconsole is, easy port forwarding, thereby enabling your friends to connect any TCP port running on your Linux system. Let’s assume you are working on a web project and it is currently accessible on your http://localhost:3000 . You can make your friends access it by forwarding port 3000 when you initiate a new session as shown.

$ teleconsole -f localhost:3000
Sample Output
Starting local SSH server on localhost...
Requesting a disposable SSH proxy on as.teleconsole.com for tecmint...
Checking status of the SSH tunnel...
Your Teleconsole ID: asce38b0cbb9db97ef16562d1feffe5b84c9a204b8
WebUI for this session: https://as.teleconsole.com/s/ce38b0cbb9db97ef16562d1feffe5b84c9a204b8
To stop broadcasting, exit current shell by typing 'exit' or closing the window.

Now when your friends join this session, they will see a message as shown.

ATTENTION: tecmint has invited you to access port 3000 on their machine via localhost:9000
Join Linux Session

Join Linux Session

They can then access your application from their browsers using the URL http://localhost:3000.

Important: Since Teleconsole is just an SSH server, any one you have shared your session ID with can request port forwarding without informing you, as shown.

$ teleconsole -f 3000:localhost:3000 join <session-id>

You can view the teleconsole help message with the following command.

$ teleconsole help

For more information, go to the Teleconsole Github repository.

That’s all! Teleconsole is a powerful SSH server to share your Unix/Linux terminal session with friends. In this article, we have explained how to use teleconsole to launch secure SSH sessions and share your terminal with friends and perform forwarding of local TCP ports.

If you have any questions or thoughts to share, use the comment form below to reach us.

FCC Retracts a Plan to Discourage Consumer Complaints

The Federal Communications Commission has reportedly dropped a proposed change in how it handles complaints that critics argued could have left consumers with fewer avenues to resolve problems with telecommunications carriers like AT&T and Verizon.

The agency is scheduled to vote Thursday on proposed changes to the complaint process, but according to the Washington Post, the most controversial changes have been removed from the draft.

The FCC offers two ways for people to complain about billing problems, privacy concerns, and other issues with telecom carriers. Formal complaints cost $225 to file and work a bit like court proceedings. But the commission also offers an informal complaint system, which is free.

Critics said that the proposed change would have left the informal complaint system toothless, forcing consumers to spend the time and money of the formal review process if they wanted to the FCC to take action on their complaints.

One reason the critics saw ill will behind the proposal: The FCC last year declined to release the full text of informal complaints it received about net neutrality ahead of the agency’s vote to jettison those rules in December. The Obama-era rules banned broadband providers from blocking or discriminating against particular internet content. The FCC highlighted the lack of formal complaints about net neutrality in support of its decision to roll back the rules, but did not address the informal complaints.

In a statement, Commissioner Jessica Rosenworcel, the FCC’s only Democratic commissioner, called the proposed change to the informal complaint process “bonkers.”

“No one should be asked to pay $225 for this agency to do its job,” she said. “No one should see Washington close its doors to everyday consumers looking for assistance in a marketplace that can be bewildering to navigate.” A spokesperson for Rosenworcel said earlier Wednesday that she was talking with other commissioners about changes to the proposal ahead of Thursday’s meeting.

On Tuesday, Representatives Frank Pallone, Jr. (D-New Jersey), and Mike Doyle (D-Pennsylvania) sent a letter to FCC Chair Ajit Pai arguing that the proposed changes would “direct FCC staff to only pass consumers’ informal complaints on to the company and then advise consumers that they file a formal complaint for a $225 fee if they are not satisfied with the company’s response.”

An FCC spokesperson told WIRED that the changes were only meant to clarify existing policy. The FCC’s website explains that it does not take action on individual informal complaints, but “the collective data we receive helps us keep a pulse on what consumers are experiencing, may lead to investigations and serves as a deterrent to the companies we regulate.”

“If the [FCC’s] Consumer and Governmental Affairs Bureau spots a troubling trend on any issue, it can refer the matter to the [FCC’s] Enforcement Bureau, which can launch a broader investigation,” the spokesman said.

But critics argue that the changes could have discouraged FCC staff from doing even that. The proposal would have removed language from the FCC’s rules specifying that the commission could contact a complainant about its “review and disposition.”

In this context, “disposition” means “resolution.” Critics of the change worried that unless the agency’s rules explicitly allow for review and action on complaints, the FCC wouldn’t have the authority or obligation to do so.


More Great WIRED Stories

Twitter Will Hide ‘Locked’ Profiles From Follower Counts

Perhaps a healthier Twitter is one with smaller follower counts—even if that comes as a blow to your ego. That’s what the company is hoping, anyway. Over the last several months, Twitter has embarked on a renewed push to fight abuse and spam, as well as encourage healthy debates and conversations, and on Wednesday the social network announced it was expanding that effort to profiles it has locked for suspicious behavior. Over the next week, it will remove these profiles from users’ follower counts.

Twitter says it’s removing locked profiles so that users’ follower counts more accurately reflect the number of real people who choose to follow their tweets. Twitter locks an account as a penalty for violating its policies or when it detects a sudden change in behavior, like a significant uptick in replies, tweeting misleading links, or if a large number of people suddenly block the account. As a security protection, Twitter also sometimes locks an account when its credentials have been posted or leaked elsewhere online.

A locked account can’t post tweets and isn’t exposed to ads until the owner verifies that everything is OK. If no one verifies the account, it remains locked. Twitter says tens of millions of locked accounts will be affected by the change, representing about 6 percent of follows across the site. Most people will lose less than four followers, but “others with larger follower counts will experience a more significant drop,” Vijaya Gadde, head of legal, policy, and trust and safety at Twitter, wrote in a blog post announcing the change. If a locked account is reclaimed by its owner, they will also regain all of their follows.

Locked accounts are different from bots and spammers; Twitter believes that most were created by real people, but as Gadde explains, Twitter “cannot confirm that the original person who opened the account still has control and access to it.” In the past, spammers and other malicious actors have taken control of accounts—especially those that are verified—in order to carry out scams or spread misinformation.

Twitter announced it was limiting the influence of locked accounts less than a week after The Washington Post reported that the social network had deleted 70 million fake or suspicious accounts in May and June. The removal of so many fake accounts may have impacted Twitter’s number of monthly active users, according to the report.

That might spell trouble for shareholders; Twitter’s share price slumped after news broke that it was purging so many fakes.

In Wednesday’s blog post, Gadde said the change to locked accounts won’t affect Twitter’s monthly or daily active user count. Locked accounts that haven’t reset their password in more than a month aren’t included in those metrics, according to the company.

Removing tens of millions of fake accounts might look like a red flag to investors, but it’s likely a welcome change for Twitter’s users. The social network has long been plagued by groups of bots and fraudulent accounts, including a notorious Russian-linked network that sowed chaos for years, culminating in an effort to influence the 2016 presidential election.

Over a year after Donald Trump took office, researchers discovered that Russian-linked bots were still tweeting. The Alliance for Securing Democracy, a group that tracks Russian disinformation efforts, says Kremlin-linked bots have tried to discredit special counsel Robert Mueller and influence the gun-control debate in the wake of mass shootings.

It’s not just Russian bad actors that Twitter has had to contend with; bots, automation, and spam have long been a part of Twitter’s DNA. The Pew Research Center found, for example, that two-thirds of links to popular websites on Twitter are posted by automated accounts. (That includes legitimate accounts that schedule tweets through an automated service.)

Follower counts have also consistently been manipulated: Comedian Joe Mande famously bought 1 million fake followers back in 2013. A New York Times investigation in January found that plenty of other celebrities and public figures have quietly purchased followers to appear more influential than they really are.

The presence of so many fake followers makes it difficult to gauge how popular a given Twitter account really is, but it also drains some of the joy from Twitter. It can be impossible to determine whether a new follower likes your tweets, or is just a bot trying to appear legitimate.

In removing locked profiles from people’s follower counts, Twitter has found a simple way to cut down on some of that noise. The company says that follower accounts may continue to change more regularly, as Twitter continues to figure out what it means to be in better health.


More Great WIRED Stories

The Rise and Fall of Uber HR Chief Liane Hornsey

In January, I sat down with Liane Hornsey, who until yesterday was Uber’s HR chief, to discuss the progress she’d made helping to reform Uber’s culture. The company had invited me to report on its turnaround, in the run-up to the release of its redesigned drivers app. But I was interested in something else: How were things at Uber since CEO Dara Khosrowshahi arrived?

She told me that she had asked an employee—a three-year veteran at Uber—how it felt to be there. “She said to me, ‘We used to feel we were good people doing good things,’” Hornsey reflected. “‘Now we feel we’re bad people doing bad things.’” She endeavored to fix that feeling.

Hornsey had arrived at Uber just a few weeks before Susan Fowler published the February 2017 blog post that propelled the company into a turmoil it is still working to recover from. She introduced herself to the staff at a teary all-hands meeting following that post. During her tenure, she oversaw numerous investigations, ran point on the Holder report, navigated through dozens of staff departures, served on the 14-person interim leadership team that ran the company after former CEO Travis Kalanick was ousted, and tried to save Uber from itself.

Yesterday evening, 18 months later, Hornsey resigned.

Hornsey’s departure is a testament to a fledgling system that empowers employees to call executives out for bad behavior.

An email she sent to employees yesterday provides no details on the reason for her departure, nor does an email Khosrowshahi sent to staff. But it arrived hours after Reuters reported that an anonymous group of Uber employees had complained that Hornsey systematically dismissed complaints of race-based discrimination. Regardless of the content of these claims, the complaint process and Hornsey’s subsequent departure is a mark of what Hornsey helped create during her time at the company—a system that allows for empowered employees to call executives out for bad behavior, and demand swift action.

Hornsey’s resignation is also a sign of what hasn’t yet been achieved. Disgruntled employees still don’t trust Uber’s systems, and they are turning to the media to air their grievances. This suggests that Khosrowshahi’s attempt to build trust among employees, an assurance that the company can address challenges internally, has not taken hold.

“We do the right thing. Period,” Khosrowshahi repeats often, by way of a mantra. And when it comes to corporate optics, there’s a prescribed list of right things. When there is a scandal, for example, someone must leave. But the actual right things—the things that honor people, advance a business, and begin to heal a fractured culture—are not always so clear. Regardless of the events that provoked it, Hornsey’s departure is a crisis for the company—a crisis that Hornsey is no longer available to manage.

To be clear, the details of what happened at Uber have yet to be released. According to Reuters, an anonymous group of employees, identifying themselves as people of color, said Hornsey made derogatory remarks about Bernard Coleman, the company’s global head of diversity and inclusion, hired in January, and disparaged and threatened Bozoma Saint John, who stepped down last month.

Reuters reports that this conflict was the reason Saint John recently left her role as chief brand officer at Uber to take a job as chief marketing officer of the talent agency Endeavor. Also, Reuters reviewed an email in which investigators from the law firm Gibson Dunn told the group of employees making the complaint that some of their allegations had been substantiated.

One thing is clear: The next person to step into the HR chief role at Uber will inherit a very different culture, thanks to the structures Hornsey built. She arrived at an organization in crisis, and one that had grown to an enormous size without having any of the human resource rails that one might expect of a company of Uber’s size. The herculean task of building out a set of systems and practices for managing the organization fell to her.

Nothing she did went beyond what industry peers were already doing. This is a company that boasted in January about newly instituted quarterly volunteer days, allowing employees to serve soup at a homeless shelter or play with hospitalized kids, as though it was something unique. Given that Uber had never compensated employees for volunteer time or demonstrated any philanthropic inklings, it was notable. Compared to other companies of its size and scale, it was notable that the program hadn’t existed before.

But most of Hornsey’s work involved building common company processes, like less biased performance reviews or management training programs. Once she’d seen the company through its initial crisis, facilitating investigations, responding directly to hundreds of emails and conducting some 200 “listening sessions,” Hornsey instituted the move designed to be her signature: pay parity for all employees, regardless of race or gender. “Very few executives want to go to this place in any company. There’s always resistance,” she told me in January. This was before Khosrowshahi started, and she’d had to sell the idea to all of the other 14 members of the executive leadership team. The larger challenge with pay parity comes well after its instituted, of course, when managers need to negotiate salaries for new hires.

Despite this transparency, Uber still got flack for refusing to release the full report commissioned in the wake of Fowler’s memo. Compiled by former attorney general Eric Holder, it resulted in 47 recommendations that were made public in June. (A second law firm was hired specifically to investigate sexual harassment after the company received 215 complaints in the wake of Fowler’s memo.) To this day, Holder’s report hasn’t been published publicly. I asked Hornsey why, given the extreme focus on transparency, the company wouldn’t publish it. “It’s totally a board decision,” she said. “Honestly, I’ve read it. I don’t know why you wouldn’t release it.”

Transparency alone, the kind that might have come from releasing that report, doesn’t fix a trust problem. As the writer Rachel Botsman, author of Who Can You Trust, has said, transparency happens once trust has been broken. What matters isn’t that you reveal everything, but that people believe you are honest with them, and that they can, in turn, be honest with you.

Ultimately, that’s why Hornsey had to resign, regardless of what comes of the investigation underway. Khosrowshahi needs a strong, experienced HR lead, but he needs his employees’ trust even more. Right now, in the face of little publicly released information and hints of a concerning investigation, leaving is what it means to do the right thing—period.


More Great WIRED Stories

How a Startup Is Using the Blockchain to Protect Your Privacy

Dawn Song, a Berkeley computer-science professor and MacArthur fellow, is a fan of cloud computing. She also thinks it needs a major rethink. “The cloud and the internet have fundamentally changed our lives mostly for good,” she says. “But they have serious problems with privacy and security—users and companies lose control of their data.”

Outsourcing data storage and processing over the internet has given companies new flexibility and consumers the power to hail rides, find dates, and socialize from a slab of glass in their pocket. The same technologies have also enabled data theft, corporate prying on our personal lives, and new forms of election manipulation.

Song says her startup, Oasis Labs, can curtail some of those problems with the help of blockchains, the new form of cryptographically secured record-keeping inspired by the digital currency bitcoin. Oasis announced $45 million in funding this week, from a mixture of big Silicon Valley VC funds and cryptocurrency investors. Song and one of her cofounders have already tested some of their ideas by helping install new privacy safeguards at Uber, the ride-hailing unicorn whose troubled past includes security incidents.

Oasis Labs cofounders, left to right: chief privacy officer Noah Johnson, chief operating officer Bobby Jaros, chief executive Dawn Song, and chief technical officer Raymond Cheng.

Elena Zhukova/Oasis Labs

In 2014, Uber was rocked by allegations that executives and employees spied on customer movements, using tools such as a map dubbed “God View.” Two years later, the company settled with New York state’s attorney general and promised to protect rider location data. Oasis grew in part from a 2017 project in which Song and two grad students, one of whom became a cofounder of the Berkeley startup, helped Uber add a more sophisticated privacy safety net.

The Berkeley researchers helped build and deploy an open source tool that limits how much employees can learn about individual customers by analyzing rider data. It’s based on a technique called differential privacy, designed to protect individuals’ identity even when data allegedly has been anonymized. It’s also used by Apple to collect data from iPhones without risking customer privacy. In Uber’s system, employees can query a database, for example, to summarize recent rides in a particular area. Behind the scenes, algorithms assess the risk that the request will leak information about individuals, and they inject random noise into the data to neutralize that risk. Ask about recent rides in a large city, and little or no noise will be needed; ask the same for a specific location, say the White House, and much more randomness will be added to obscure traces that might represent specific individuals.

Uber’s differential privacy software doesn’t use a blockchain, a kind of digital records system guarded by cryptography that can limit and log who makes changes or additions. Song says privacy and security systems can be much stronger if they do. We must take Uber’s word that the company has deployed its differential privacy system correctly, for example. Companies that build privacy or security systems plugged into Oasis Labs’ blockchain will be able to provide cryptographic assurances to one another, or their customers, that their systems are doing what was promised, says Song. She describes Oasis Labs as trying to provide the security and privacy infrastructure that the internet is lacking—and for which we are paying a price.

Oasis Labs’ platform can also host small programs, dubbed smart contracts, which can mediate transactions between different people or companies. That makes it similar to ethereum, the second most valuable cryptocurrency system. But Oasis Labs’ blockchain is specifically designed to enable security applications and builds on Berkeley research that Song says makes the system more scalable and practical than existing blockchains.

Song says her company is talking with organizations in healthcare, finance, and ecommerce hoping to make use of the Oasis platform when it fully launches, probably next year. One large ecommerce company is interested in building internal privacy controls like Uber’s, for example, and sharing more supply chain data with partners while protecting commercially sensitive information. Another project is building a way for health patients to donate medical data for machine learning research. Oasis’ technology will be used to assure patients that their data can’t be diverted for other uses. For the ultra-paranoid, Oasis plans to integrate its software with an open source security chip in development by Berkeley and MIT, similar to the chip that underpins the iPhone’s security, to protect crucial smart contracts against meddling.

Oasis Labs is launching at a time when blockchain projects have some credibility issues. The total value of all cryptocurrencies is estimated at more than $250 billion, and venture investors sank more than $1 billion into blockchain startups last year, according to CB Insights. But cryptocurrencies aren’t widely used and are blighted by scammers, thieves, and poor security. Corporate blockchain projects have delivered more hot air and hype than practical technology.

Song says the hype around blockchain and lurid adventures of some devotees distract from a genuine opportunity to remake the fundamentals of how computers serve us. “People shouldn’t throw the baby out with the bathwater,” she says.

Christian Catalini, a professor at MIT Sloan School of Management, says Song isn’t the only computer science deep thinker who feels that way. “More talent from academia has been moving into the market,” he says. “It’s a new trend.” Researchers from Johns Hopkins and MIT, for example, are contributing to the development of ZCash, a cryptocurrency designed to offer fully anonymous digital transactions, something Bitcoin does not. JP Morgan is partnering with the project, saying anonymity could help companies keep their finances more private. Last year professors from Cornell and Northwestern launched a startup called bloXroute Labs to make cryptocurrencies more scalable, a problem that has plagued bitcoin and ethereum.

Catalini believes the profs rubbing elbow patches with blockchain fanatics could help realize more of the young industry’s dreams. Still, with funded projects ranging from efforts to offer better marine insurance to “cryptoassets” claimed to fight climate change, probably not all will work. “The vast majority of these projects will fail,” Catalini guesses. “But in phases of extreme technological experimentation, people explore in many different directions, and that’s good for society in the long run.”


More Great WIRED Stories