https://www.friendbookmark.com/photos/category/new/36/ Most recent added photos in the category of https://www.friendbookmark.com/videos/1058/angular-ventures-coronavirus-impact-surveyIn an effort to support the community of early-stage founders, Angular Ventures surveyed 128 founders in Europe, Israel, and the United States. We asked them how the Coronavirus has impacted their businesses and how they are responding. This report is the result of that work. Topics covered in the presentation slides: 1. May 7, 2020 Coronavirus: Impact & Response A global survey of early-stage technology founders 2. May 7, 2020 2 Gil Dibner Partner gil@angularventures.com @gdibner LON/TLV Anne Blum Head of Platform anne@angularventures.com @anneablum NYC Andrew Poesaste Associate andrew@angularventures.com @poetential LON Angular Ventures is a venture capital fund based in London, Tel Aviv, and New York that backs early-stage enterprise tech companies from Europe and Israel. We typically invest between $250K and $1.5M anywhere from âday zeroâ to Series A. 3. May 7, 2020 3 âI believe, or perhaps hope, that the VC industry will take what I consider to be a more mature approach to financing startups, with less talk of unicorns and more focus on making ventures economically sustainable. Those ventures with the capacity for stunning valuations will retain that capacity, but all ventures will carry lower risk as a result.â CEO, UK proptech company âMy concern is the influence on 2021-2022 customers budgets and priorities. We have been growing organically 2.5x each year in the past 3 years, we now expect this growth to be slowed.â CEO, Israeli drone company âCorporates and start-ups will adjust to this new normal. Selling over video-calls already worked, but now is necessary. Same for fundraising. The faster everybody adjust to the new normal, the less impact Covid19 will have to the economy. Big corporates are all saying: it's business as usual but adjusted to the new normal.â CEO, Dutch SaaS company âOur current strategy is to stay alive by any means possible and be top of mind for when customers are ready to buy again.â CEO, Romanian edtech company âOverall, I'm cautiously optimistic about our chances of survival, itâs looking likely we could thrive. If we can raise sometime in the next 6 months, I believe we could dominate the space.â CEO, UK transportation company In an effort to support the community of early-stage founders, Angular Ventures surveyed 128 founders in Europe, Israel, and the United States. We asked them how the Coronavirus has impacted their businesses and how they are responding. This report is the result of that work. 4. May 7, 2020 4 Founder, 97% Survey demographics 34% 30% 17% 14% 5% 1% 1-5 6-10 11-20 21-50 51-100 Over 100 Number of employees Europe, 37% UK, 29% Israel, 23% USA, 6% Other, 5% 84% 4% 2% 2% 2% 2% 5% CEO CTO COO VP Eng. VP Mktg / CMO VP Sales / CRO Other Job title 16% 23% 36% 20% 2% 4% Pre-revenue Early POC revenue Less than $1M/year $1M-$5M/year $5M-$10M/year $10M-$25M/year Revenue level 12% 34% 39% 11% 3% 2% None Pre-seed / angels Seed Series A Series B or later Other Funding stage 5. May 7, 2020 5 Contents 1 2 3 4 5 6 Concern & resilience Fundraising & runway Top-line impact Product considerations Marketing strategy Human capital 45% of founders are not confident their startup will survive the recession 53% of companies have 12 months or less of runway 58% of founders are already seeing a negative impact from Coronavirus on sales 47% of founders are changing their product offering in response to Coronavirus 55% of companies are eliminating or reducing the marketing spend 51% of companies are implementing pay cuts 6. May 7, 2020 6 Coronavirus Impact Survey: Concern & resilience 7. May 7, 2020 7 Coronavirus Impact Survey Fears & concerns â As a group, most founders are more concerned about the impact of the pandemic on the economy (81%) than they are concerned about the virus itself (13%) â 55% of founders we surveyed are confident their company will survive the recession â 9% of founders are very concerned that their company wonât survive the recession 81% 13% 6% More concerned about COVID-19 or its impact on the economy COVID-19's impact on the economy COVID-19 Neither 55% 37% 9% Concern of startup's survival during recession Not worried, startup will survive this recession Somewhat worried startup won't survive this recession Very worried startup won't survive this recession 8. May 7, 2020 8 Coronavirus Impact Survey Does revenue buy resilience? â Early stage companies that are pre-revenue or in early POC stages seem to be the most concerned about survival â There is also a high level of concern among substantially more mature companies, perhaps due to a higher cost basis âWe're pre-product market fit. We can't iterate and improve without customers and feedback. Any leads we had, have gone cold. I think this has killed us.â - CEO of a UK SaaS pre-seed company 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Pre-revenue Early POC revenue Less than $1M/year $1M-$5M/year $5M-$10M/year $10M-$25M/year Concern level by revenue level Not worried, startup will survive this recession Somewhat worried startup won't survive this recession Very worried startup won't survive this recession 9. May 7, 2020 9 Coronavirus Impact Survey Most common concerns â Out of the major concerns listed, fundraising was the most frequently cited (58%) by founders â Declining sales (46%) and changing customer priorities (38%) completed the top three concerns âWe were just about to close our pre-seed round when the virus arrived. All of our investors went cold and have pushed conversations with us out by 4-6 months.â - CEO of an Irish analytics company 58% 46% 39% 25% 15% 9% 9% 3% Fundraising in this environment and running out of cash Declining sales Customers deprioritizing startup's solution Team moral Reduction in employee productivity Pressure to reduce pricing Disappearance of marketing channels Layoffs Top concerns (up to three selected) 10. May 7, 2020 10 Coronavirus Impact Survey Back to normal? â 43% of survey respondents believe things will be back to normal by the end of 2020 â 59% of founders expect things will be back to normal by March of 2021 â 11% of founders believe things will never return to normal âWhile I expect in one year life will feel normal again, I expect the recession to take at least two years to overcome.â - CEO of an American hospitality company 1% 21% 21% 16% 13% 7% 7% 4% 11% April to June 2020 July to September 2020 October to December 2020 January to March 2021 April to June 2021 July to September 2021 October to December 2021 Sometime in 2022 or later Never Expectation for when things will "return to normal" 11. May 7, 2020 11 Coronavirus Impact Survey Optimism is not equally distributed â UK founders are the most pessimistic - only 48% stating things will return to normal by March 2021 â American founders are the most optimistic 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% UK Europe Israel USA Expectation for when things "return to normal" by region Never Sometime in 2022 or later October to December 2021 July to September 2021 April to June 2021 January to March 2021 October to December 2020 July to September 2020 April to June 2020 12. May 7, 2020 12 Coronavirus Impact Survey: Fundraising & runway 13. May 7, 2020 13 Coronavirus Impact Survey Fundraising slowdown â Venture investment in Europe and Israel in 2020 started at the highest pace on record for January and February â both were record months â In March, as the effects of Coronavirus became clear, the pace of investment started to drop off and dipped below last yearâs levels 3,489 2,842 2,870 3,037 3,560 4,394 3,889 2,852 2,831 3,651 3,583 1,904 3,755 3,398 2,436 2,611 1,000 1,500 2,000 2,500 3,000 3,500 4,000 4,500 5,000 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Number of VC Investments (US $M, Europe & Israel) 2019 2020 Source: Angular Ventures Data 14. May 7, 2020 14 Coronavirus Impact Survey VC deals keep steady â Although the total investment amounts slowed in March and April, the number of deals done actually remains above last yearâs levels Source: Angular Ventures Data 182 157 154 184 194 197 218 103 194 228 180 152 197 213 174 194 100 120 140 160 180 200 220 240 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Number of VC Investments (Europe & Israel) 2019 2020 15. May 7, 2020 15 0 50 100 150 200 250 0 500 1,000 1,500 2,000 2,500 3,000 3,500 4,000 4,500 5,000 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr 2017 2018 2019 2020 US$M Total VC Investment Volume (Europe & Israel) Total volume of investments (US $M) Number of Investments Viewed over a longer time horizon, the first part of 2020 is still looking relatively strong. This picture does not match what many founders are seeing in the field. Why not? Part of the reason is the long lead time. Investments concluded in March or April may have been sourced in January or February, well before the scope of the Coronavirus slowdown became clear Source: Angular Ventures Data Source: Angular Ventures Data 16. May 7, 2020 16 Coronavirus Impact Survey Into a new era â 1Q20 saw the lowest annual growth rate in quarterly VC investment volume across Europe & Israel in over three years â This comes after two distinct periods of acceleration (in 2017 and 2019) â VC investment in April 2020 was down 14% from April 2019, a figure that doesnât bode well for 2Q20 activity 9.3% 40.1% 79.7% 39.8% 43.2% 12.7% 11.2% 32.5% 57.7% 69.2% 53.7% 40.4% 3.5% -14.0% 1Q17 2Q17 3Q17 4Q17 1Q18 2Q18 3Q18 4Q18 1Q19 2Q19 3Q19 4Q19 1Q20 April 20 VC Investment, YoY growth rate by quarter (Europe & Israel, US $M) Source: Angular Ventures Data 17. May 7, 2020 17 Coronavirus Impact Survey Runway is everything â Roughly half of early stage companies surveyed have 12 months or less of runway, not surprising given that most raise for 18 months â 37% have 12-24 months of runway â Only 8% are cash-flow positive 25% 28% 17% 20% 2% 8% 1 to 6 months of runway 7 to 12 months of runway 13 to 18 months of runway 19 to 24 months of runway More than 2 years of runway Positive cash flow Current runway (in months) 18. May 7, 2020 18 Coronavirus Impact Survey Runway drives perceived risk â Not surprisingly, shorter runways lead to higher levels of concern â Concern among positive cash flow companies may reflect companies operating on thin margins âI largely think the crisis will make life harder for startups. Having said that, I think the startups that will be most affected are startups that have immediate cash flow concerns and that are in a scaling phase.â - CEO of a UK construction pre-seed company 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 1 to 6 months of runway 7 to 12 months of runway 13 to 18 months of runway 19 to 24 months of runway More than 2 years of runway Positive cash flow Concern level by runway Very worried startup won't survive this recession Somewhat worried startup won't survive this recession Not worried, startup will survive this recession 19. May 7, 2020 19 Coronavirus Impact Survey Runway lengthening is an active process â Companies with longer runways typically engaged in aggressive cost-cutting to achieve that runway âAs an early stage startup in high corona impact industry, we have to extend our runway as much as possible. We made cuts that would allow us to run for 24 months with only hands on staff.â - CEO of an American hospitality company 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 1 to 6 months of runway 7 to 12 months of runway 13 to 18 months of runway 19 to 24 months of runway More than 2 years of runway Positive cash flow Cost cutting by runway No costs cut Cut costs but no meaningful impact on runway Cut costs and added 1-3 months of additional runway Cut costs and added 3-6 months of additional runway Cut costs and added 7-9 months of additional runway Cut costs and added 10-12 months of additional runway Cut costs and added more than 12 months of additional runway 20. May 7, 2020 20 Coronavirus Impact Survey Big rounds help â 54% of Series A companies report 18+ months of runway â For seed companies, that drops to about 36% â Only 16% of pre-seed companies have over 18 months of runway â 66% of pre-seed companies have 12 months or less of runway, and 40% have less than six months 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% None Pre-seed / angels Seed Series A Runway (months) by last round of financing Positive cash flow More than 2 years of runway 19 to 24 months of runway 13 to 18 months of runway 7 to 12 months of runway 1 to 6 months of runway 21. May 7, 2020 21 Coronavirus Impact Survey Extending runway â Across our group of respondents, only 13% have not taken any actions to extend runway 13% 52% 46% 43% 30% 23% 17% 15% 14% 8% 6% No changes made or planned Reduced marketing spend Hiring freeze Reduced or eliminated office space/rent Significant changes to the product roadmap and timing Across-the-board salary reductions Significant product changes Termination of service provider contracts Employee furloughs or unpaid leave Increased use of outsourcing Employee layoffs Actions taken to extend runway 22. May 7, 2020 22 Coronavirus Impact Survey Modest impact of cost cutting for most â 88% of companies surveyed have cut costs â About a quarter of those that cut costs failed to achieve any meaningful impact on their cash runway â 11% of companies have been able to cut costs to achieve more than 6 months of additional runway 18% 20% 28% 23% 5% 3% 3% No costs cut Cut costs but no meaningful impact on runway Cut costs and added 1- 3 months of additional runway Cut costs and added 3-6 months of additional runway Cut costs and added 7-9 months of additional runway Cut costs and added 10-12 months of additional runway Cut costs and added more than 12 months of additional runway Cost cutting to extend runway 23. May 7, 2020 23 Coronavirus Impact Survey Fundraising strategy â 17% of companies have begun fundraising immediately to ensure sufficient cash â 28% of companies have delayed their fundraising plans in response to the pandemic âWeâre delaying fundraising and focusing on being break-even.â - CEO of an Israeli SaaS Series A company 17% 3% 5% 46% 17% 11% Beginning fundraising immediately to ensure sufficient cash Brought fundraising earlier by more than six months Brought fundraising earlier by less than six months Fundraising strategy has not changed Pushed fundraising back by less than six months Pushed fundraising back by more than six months Change in fundraising strategy 24. May 7, 2020 24 Coronavirus Impact Survey: Top Line Impact 25. May 7, 2020 25 Coronavirus Impact Survey ACVs â Companies in the survey had a wide range of ACVs (annual contact values): from under $100/year to over $500K/year â The most common ACV range was between $10K/year and $100K/year â We also asked founders about their go-to-market methodology and assigned each company to one of five categories 7% 10% 21% 42% 16% 4% Less than $100 per year Between $100 and $1,000 per year Between $1,000 and $10,000 per year Between $10,000 and $100,000 per year Between $100,000 and $500,000 per year Over $500,000 per year Average contract value 29% 25% 23% 19% 5% Self-service Inside sales Field sales Mixed (Inside & Field) Channel / Indirect Go-To-Market models used 26. May 7, 2020 26 Coronavirus Impact Survey Impact on sales â As expected, the majority of respondents (58%) reported that Coronavirus has negatively impacted sales â 19% reported that Coronavirus has had a positive impact on their sales âCustomers don't want to pay right now even for vital services like rent, so they definitely aren't even thinking to pay for SaaS services.â - CEO of an American hospitality SaaS company 8% 11% 23% 31% 27% Sales have drastically increased Sales have slightly increased Sales have not changed Sales have slightly decreased Sales have drastically decreased Impact on sales 27. May 7, 2020 27 Coronavirus Impact Survey Sales cycle â Many respondents reported some negative impacts on sales cycles â 71% of respondents reported either a partial or significant slowdown in sales cycles â 23% reported that customers are âghostingâ them â 20% reported that budget priorities seem to be shifting away from their offering âOur entire pipeline of enterprise customers is frozen.â - CEO of a Romanian EdTech company 23% 36% 35% 20% 5% 15% 23% Some customers are disengaging or ghosting Significant slowdown in aales cycles Some slowdown in sales cycles Deprioritization of product Some key customer contacts have been laid off No impact on sales cycles Some customers are now more available Impact on sales cycles 28. May 7, 2020 28 Coronavirus Impact Survey Churn & pricing â 66% of respondents reported that Coronavirus has not yet had an impact on churn â In the two months since the Coronavirus emerged, nearly a quarter of companies have seen churn increase â 36% of companies reported some pricing pressure â 5% reported âsignificantâ pricing pressure 3% 8% 66% 20% 3% Churn has drastically decreased Churn has slightly decreased Churn has not changed Churn has slightly increased Churn has drastically increased Impact on churn 5% 31% 64% Significant pushback on pricing Some moderate pushback on pricing from some accounts No pricing pressure Pricing pressure 29. May 7, 2020 29 Coronavirus Impact Survey Sales motion matters â Companies with strong channel/indirect sales motions are showing more resilience in sales âHardware startups need face-to-face meetings! Zoom doesnât cut it for us.â - CEO of an Israeli AgTech company 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Self-service Inside sales Mixed (Inside & Field) Field sales Channel / Indirect Sales by primary sales motion Sales have drastically increased Sales have slightly increased Sales have not changed Sales have slightly decreased Sales have drastically decreased 30. May 7, 2020 30 Coronavirus Impact Survey Sales motion & churn â Reports of increased churn seem concentrated in companies with sales motions that emphasize self-service or field sales â This may reflect the difficulty of rapidly spinning up an inside sales operation which â today â is the only way to effectively engage with customers 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Self-service Inside sales Mixed (Inside & Field) Field sales Channel / Indirect Churn by primary sales motion Churn has drastically decreased Churn has slightly decreased Churn has not changed Churn has slightly increased Churn has drastically increased 31. May 7, 2020 31 Coronavirus Impact Survey Churn concentrated at mid-range prices â While many companies are reporting moderate increases in churn, reports of dramatic increases in churn seem highly concentrated at the $1,000 to $10,000 ACV price point â This price point is large enough to get noticed (and cut) by distressed customers, but not high enough to signal deep customer commitment 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Less than $100 per year Between $100 and $1,000 per year Between $1,000 and $10,000 per year Between $10,000 and $100,000 per year Between $100,000 and $500,000 per year Over $500,000 per year Churn by ACV Churn has drastically decreased Churn has slightly decreased Churn has not changed Churn has slightly increased Churn has drastically increased 32. May 7, 2020 32 Coronavirus Impact Survey More money, more problems â The higher the ACV, the more likely a company is to report pricing pressure â Interestingly, there is also significant pricing pressure being reported by the sub $100/year ACV companies 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Less than $100 per year Between $100 and $1,000 per year Between $1,000 and $10,000 per year Between $10,000 and $100,000 per year Between $100,000 and $500,000 per year Over $500,000 per year Pricing pressure by ACV No pricing pressure Some moderate pushback on pricing from some accounts Significant pushback on pricing 33. May 7, 2020 33 Coronavirus Impact Survey: Product evolution in response to crisis 34. May 7, 2020 34 Coronavirus Impact Survey Product changes â 47% of early stage companies are taking steps to change their product offerings in response to the Coronavirus âWe added cheaper, more self-service oriented options to our original enterprise only product.â - CEO of a German FinTech company 53% 26% 21% Product has not changed Changes to the product are planned Product has already changed Product changes 35. May 7, 2020 35 Coronavirus Impact Survey Everyone is changing their products â Companies of all stages are adjusting their products â Itâs notable and potentially surprising that Series A+ companies are changing their products more aggressively than their earlier stage peers â potentially due to more experienced product teams and more engineering resources 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% None Pre-seed / angels Seed Series A Series B or later Product changes Product has not changed Changes to the product are planned Product has already changed 36. May 7, 2020 36 Coronavirus Impact Survey Specific product changes â The most frequent product changes involve improving self- service onboarding, reflecting an era where customer support is more challenging than ever â Some companies are also adding features to support remote work or new markets, such as healthcare â 6% of companies are âcompletely pivotingâ their product 6% 15% 19% 25% 46% Completely pivoted the product Added features to support remote work Changes in product to support targeting a new market (e.g. healthcare) Improved self-service onboarding No planned changes to the product Product changes 37. May 7, 2020 37 Coronavirus Impact Survey: Marketing in a time of crisis 38. May 7, 2020 38 Coronavirus Impact Survey Marketing budgets â More than half of respondents indicated that they are eliminating or reducing their marketing budget in response to Coronavirus âWe cut off all paid marketing and are only doing things for free, and only online (webinars, blog posts, partnerships).â - CEO of an Israeli enterprise SaaS company 14% 31% 3% 39% 13% Marketing budget has been increased Marketing budget has remained the same Marketing budget reductions are planned or likely, but haven't happened yet Marketing budget has been reduced Marketing budget has been eliminated Change in marketing budget 39. May 7, 2020 39 Coronavirus Impact Survey Marketing is changing â In addition to reducing marketing spend, most companies (63%) have changed their marketing content to reflect the situation â Only 26% of companies report no change in marketing content âWe are changing our message to be supportive in the situation and not product pushing at all.â - CEO of an oil & gas SaaS company 26% 63% 11% Marketing content has remained the same Marketing content has changed to reflect the current situation Marketing has been put on pause for now Change in marketing content 40. May 7, 2020 40 Coronavirus Impact Survey Marketing stories â Companies are more likely to put their marketing efforts on pause if their sales are negatively impacted by Coronavirus â Companies that have dramatically increased sales are changing their content to reflect the current situation âAs some segments are moving slower, we can invest more time on marketing and content to have a stronger playbook when deals start moving again.â - CEO of an Estonian HR SaaS company 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Sales have drastically decreased Sales have slightly decreased Sales have not changed Sales have slightly increased Sales have drastically increased Impact of sales on marketing strategy Marketing content has remained the same Marketing content has changed to reflect the current situation Marketing has been put on pause for now 41. May 7, 2020 41 Coronavirus Impact Survey Marketing agility â The higher the ACV, the more likely it is that a company has reduced its marketing spend â This may be due to increased market uncertainty coupled with long sales cycles âI think my market is going to sleep now and it's stupid to waste money trying to wake the dead.â - CEO of a German enterprise AI company 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Less than $100 per year Between $100 and $1,000 per year Between $1,000 and $10,000 per year Between $10,000 and $100,000 per year Between $100,000 and $500,000 per year Over $500,000 per year Marketing spend by ACV Marketing budget has been increased Marketing budget has remained the same Marketing budget reductions are planned or likely, but haven't happened yet Marketing budget has been reduced Marketing budget has been eliminated 42. May 7, 2020 42 Coronavirus Impact Survey: Tough HR decisions 43. May 7, 2020 43 Coronavirus Impact Survey Hiring strategy â 56% of respondents indicated that they have either scaled back hiring plans or implemented a complete hiring freeze âWeâre implementing a hiring freeze and cutting (almost) all consulting and purchased services.â - CEO of an Estonian HR SaaS company 10% 34% 9% 9% 38% Increased the number of hires planned Hiring strategy has not changed Reduced the number of hires planned by up to 25% Reduced the number of hires planned by more than 25% Implemented a complete or near-complete hiring freeze Impact on hiring strategy 44. May 7, 2020 44 Coronavirus Impact Survey Hiring by runway â Companies with less runway are more likely to freeze hiring 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 1 to 6 months of runway 7 to 12 months of runway 13 to 18 months of runway 19 to 24 months of runway More than 2 years of runway Positive cash flow Hiring strategy by runway Increased the number of hires planned Hiring strategy has not changed Reduced the number of hires planned by more than 25% Reduced the number of hires planned by up to 25% Implemented a complete or near-complete hiring freeze 45. May 7, 2020 45 Coronavirus Impact Survey Hiring plans by region â American early stage companies have been the most aggressive in implementing hiring freezes with 50% choosing to do so â Israeli early stage companies are the most aggressive in terms of potential future hiring âWe have accelerated hiring of key staff as appropriate candidates suddenly became available in the market.â - CEO of a UK construction company USA UK Europe Israel Hiring plans by region Increased the number of hires planned Hiring strategy has not changed Reduced the number of hires planned by up to 25% Reduced the number of hires planned by more than 25% Implemented a complete or near-complete hiring freeze 46. May 7, 2020 46 Coronavirus Impact Survey Pay cuts â Half of survey respondents are implementing pay cuts â Nearly a quarter indicated that across-the-board pay cuts have already taken place âWeâre doing pay cuts to lengthen our positive cash flow capability while not firing any employees. All employees accepted the situation with understanding.â - CEO of an Israeli drone company 49% 15% 1% 13% 23% No pay cuts are planned Pay cuts are planned or likely, but haven't happened yet Only employees have taken a pay cut Only founders have taken a pay cut All founders and employees have taken a pay cut Pay cuts implemented 47. May 7, 2020 47 Coronavirus Impact Survey Pay cuts vary â Where companies are implementing pay cuts, they are most frequently between 10% and 20% â Some companies, however, are implementing far more severe pay cuts in an effort to survive the crisis â 30% of companies that have implemented pay cuts have cut pay by over 40% 5%6% 1% 12% 14% 62% More than 50%Between 40% and 50% Between 30% and 40% Between 20% and 30% Between 10% and 20% No pay cuts have been made Pay cut amounts 48. May 7, 2020 48 Coronavirus Impact Survey Layoffs â 76% of companies are not planning to lay off any staff â The companies that have conducted layoffs have mostly laid off less than 30% of their workforce 76% 10% 14% No one has been laid off, layoffs are not likely to happen Layoffs are planned or likely, but haven't happened yet Team members have already been laid off Companies planning layoffs 46% 42% 13% Between 0% and 10% Between 10% and 30% More than 30% Percentage of workforce laid off 49. May 7, 2020 49 Coronavirus Impact Survey Layoffs by runway expansion â Layoffs have been a key driver of increases in runway â 50% of companies which added 7-12 months of runway have already conducted layoffs 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% No costs cut Cut costs but no meaningful impact on runway Cut costs and added 1-3 months of additional runway Cut costs and added 3-6 months of additional runway Cut costs and added 7-9 months of additional runway Cut costs and added 10-12 months of additional runway Cut costs and added more than 12 months of additional runway Layoffs by runway expansion No one has been laid off, layoffs are not likely to happen Layoffs are planned or likely, but haven't happened yet Team members have already been laid off 50. May 7, 2020 50 Coronavirus Impact Survey Layoffs by region â Overall, Americans and Israelis have been the most aggressive in implementing and planning layoffs â European founders have been the least aggressive â This may be partially cultural, but may also reflect strong government support policies in European countries 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% USA Israel UK Europe Layoffs by region No one has been laid off, layoffs are not likely to happen Layoffs are planned or likely, but haven't happened yet Team members have already been laid off 51. May 7, 2020 For more resources, visit www.angularventures.com and subscribe to our weekly newsletter. Â2020 by Angular UK Opco Ltd. In compliance with applicable UK regulations, Angular Ventures is an Appointed Representative of Sapia Partners LLP (550103), a firm authorised and regulated by the Financial Conduct Authority (FCA). https://www.friendbookmark.com/videos/984/docker-enables-devopsSome tools such as Chef and Jenkins are used by engineers in ops to great effect. Rarely though, a technology brings a paradigm to the masses. Docker, like cloud virtualization is of this more rare breed. Topics covered: 1. Docker Enables DevOps Boyd E. Hemphill @behemphi @stackengine 2. History Started Austin DevOps In 2012 3. History Started Austin DevOps In 2012 At Feedmagnet, Chef saved my bacon learned I was âdoing DevOpsâ at Chef Conf 4. History Started Austin DevOps In 2012 At Feedmagnet, Chef saved my bacon learned I was âdoing DevOpsâ at Chef Conf Our first host and sponsor was CopperEgg 5. History Started Austin DevOps In 2012 At Feedmagnet, Chef saved my bacon learned I was âdoing DevOpsâ at Chef Conf Our first host and sponsor was CopperEgg After moving from a tools focus to philosophy and models have grown to 700 members 6. History Started Austin DevOps In 2012 At Feedmagnet, Chef saved my bacon learned I was âdoing DevOpsâ at Chef Conf Our first host and sponsor was CopperEgg After moving from a tools focus to philosophy and models have grown to 700 members Ended up at StackEngine when the CopperEgg founders started this venture 7. What is The Goal of Your Company? 8. What is The Goal of Your Company? Make Money! 9. So â What is DevOps? 10. Is DevOps a Process? 11. Is it an intersection of overlapping concerns? 12. Is DevOps a Culture? 13. So â What is DevOps? DevOps is a Philosophy 14. So â What is DevOps? DevOps is a Philosophy All of the previous are models for implementation 15. DevOps: DevOps is the way in which a technology organization embeds itself in a business to the benefit of that business. 16. Business Basics Profit 17. First Principles Profit Business Value 18. Profit, Revenue & Cost Profit = Revenue - Cost 19. Profit, Revenue & Cost Profit = Revenue - Cost Drive Cost to $0 and you are out of business 20. Profit, Revenue & Cost Profit = Revenue - Cost Drive Cost to $0 and you are out of business Increasing Revenue has no theoretical cap 21. Tools vs. Technology Tools have their greatest impact on cost 22. Tools vs. Technology Tools have their greatest impact on cost Tools are the result of implementing a DevOps model 23. Tools vs. Technology Tools have their greatest impact on cost Tools are the result of implementing a DevOps model Technology enables revenue creation 24. Tools vs. Technology Tools have their greatest impact on cost Tools are the result of implementing a DevOps model Technology enables revenue creation Technology enables the creation of new DevOps models. 25. Tools v. Tech Virtualization Configuration Management Continuous Integration Continuous Delivery Service Discovery Containers Vmware, AWS, Heroku CFEngine, Puppet, Chef, Ansible Go, Hudson, Jenkins, Travis Artifactory, Nexus, Shippable Zookeeper, etcd, consul (no SaaS yet) FreeBSD Jails, LXC, Docker 26. Ideally We do ourselves a disservice by naming technology with tools. 27. Ideally We do ourselves a disservice by naming technology with tools. We should be talking about âsolving a config management problem,â not âwriting Chef codeâ 28. Realistically Good tools enable a technology to be consumed by mere mortals 29. Realistically Good tools enable a technology to be consumed by mere mortals CFEngine has been around a long time, but Puppet and Chef raised the config management conversation 30. Realistically Good tools enable a technology to be consumed by mere mortals CFEngine has been around a long time, but Puppet and Chef raised the config management conversation VMware is world class virtualization, but AWS brought virtualization to the masses. 31. Realistically Good tools enable a technology to be consumed by mere mortals CFEngine has been around a long time, but Puppet and Chef raised the config management conversation VMware is world class virtualization, but AWS brought virtualization to the masses. Twitter, Facebook, Google, Pantheon have all be using containers for some years. Docker brings containers to conversations to all phases of the SDLC 32. Docker - Opportunity & Consequence Density Factoring Build and Test System Architecture 33. Density 34. Density - Defined The amount of idle compute on a host tends to zero 35. Density - Benefits 36. Density - Benefits Reduces VM consumption thus reducing cost 37. Density - Benefits Reduces VM consumption thus reducing cost Reduces power consumption in a physical setting 38. Density - Concerns 39. Density - Concerns Fewer VMs in fewer physical locations 40. Density - Concerns Fewer VMs in fewer physical locations Location of VMs or Hardware critically important 41. Density - Concerns Fewer VMs in fewer physical locations Location of VMs or Hardware critically important Spare capacity on hosts not there to save you during usage spikes 42. Density - Concerns Fewer VMs in fewer physical locations Location of VMs or Hardware critically important Spare capacity on hosts not there to save you during usage spikes YACL - Yet another complexity layer: containers on vms on hardware 43. Density - Concerns Fewer VMs in fewer physical locations Location of VMs or Hardware critically important Spare capacity on hosts not there to save you during usage spikes YACL - Yet another complexity layer: containers on vms on hardware Container Sprawl 44. Density - Business 45. Density - Business Reduces VM consumption thus reducing cost 46. Density - Business Reduces VM consumption thus reducing cost Helpful by not enough to merit the difficulty of a migration 47. Density - Adoption 48. Density - Adoption Purely a production concern 49. Density - Adoption Purely a production concern Discussed a great deal, but implementation implications too large 50. Density - Adoption Purely a production concern Discussed a great deal, but implementation implications too large Revolution, not evolution 51. Density - Adoption Purely a production concern Discussed a great deal, but implementation implications too large Revolution, not evolution Tools just not there yet 52. Density - Tools 53. Density Tools Gap Scheduling that is location aware - bin packing problem 54. Density Tools Gap Scheduling that is location aware - bin packing problem 55. Density Tools Gap Scheduling that is location aware - bin packing problem Inventory management images containers hosts 56. Density Tools Available StackEngine Tutum Fleet Dies Control Center Docker Red Hat Google AWS â 57. Factoring Distributed Applications 58. Factoring - Defined Reduce your production topology to a single machine 59. Factoring - Defined Reduce your production topology to a single machine Works great for many applications 60. Factoring - Defined Reduce your production topology to a single machine Works great for many applications Vagrant is a killer tool 61. Factoring - Benefits 62. Factoring - Benefits Vagrant multi-machine is resource hungry. Run a single VM with multiple containers 63. Factoring - Benefits Vagrant multi-machine is resource hungry. Run a single VM with multiple containers Developer, not Ops, driven 64. Factoring - Benefits Vagrant multi-machine is resource hungry. Run a single VM with multiple containers Developer, not Ops, driven Developers need not learn config management, only Dockerfile 65. Factoring - Concerns 66. Factoring - Concerns Impedence: How do Build, QA and Ops teams become aware of config change 67. Factoring - Concerns Impedence: How do Build, QA and Ops teams become aware of config change Does Dockerfile have enough power 68. Factoring - Concerns Impedence: How do Build, QA and Ops teams become aware of config change Does Dockerfile have enough power Is it necessary, or just cool? (sharding) 69. Factoring - Business 70. Factoring - Business Unclear 71. Factoring - Business Unclear Could speed up development, but is only a local optima 72. Factoring - Adoption 73. Factoring - Adoption By far the most common adoption path 74. Factoring - Adoption By far the most common adoption path Typically seen in shops where Vagrant perceived as complex 75. Factoring - Adoption By far the most common adoption path Typically seen in shops where Vagrant perceived as complex Often gains traction in Build/QA 76. Factoring - Tools 77. Factoring - Tools Gap Application modeling simplification 78. Factoring - Tools Gap Application modeling simplification Workflow management 79. Factoring - Tools Available Boot2Docker Fig Vagrant Docker 80. Build and Test Grids 81. Build and Test Grids - Defined Testing a number of language versions and environments in parallel 82. Build and Test Grids - Defined Testing a number of language versions and environments in parallel Very important to installed software 83. Build and Test Grids - Defined Testing a number of language versions and environments in parallel Very important to installed software Example Testing on Centos 6.5, Ubuntu 14.04 and CoreOs, with the last three stable Docker releases 84. Build and Test Grids - Benefits 85. Build and Test Grids - Benefits Containers come up fast making for shorter builds 86. Build and Test Grids - Benefits Containers come up fast making for shorter builds Multiple containers on a build agent improves density 87. Build and Test Grids - Benefits Containers come up fast making for shorter builds Multiple containers on a build agent improves density Makes it possible to test many more permutations of system environments 88. Build and Test Grids - Benefits Containers come up fast making for shorter builds Multiple containers on a build agent improves density Makes it possible to test many more permutations of system environments Potential for more build parallelism 89. Build and Test Grids - Concerns 90. Build and Test Grids - Concerns Is a container based test environment close enough to production 91. Build and Test Grids - Concerns Is a container based test environment close enough to production Impedance: how does the container get from build or test environment to production 92. Build and Test Grids - Business 93. Build and Test Grids - Business Increased grid density reduces costs 94. Build and Test Grids - Business Increased grid density reduces costs Reducing build times increase innovation 95. Build and Test Grids - Business Increased grid density reduces costs Reducing build times increase innovation Reducing build times increase development velocity 96. Build and Test Grids - Business Increased grid density reduces costs Reducing build times increase innovation Reducing build times increase development velocity Increase test speed keeps QA from becoming a bottleneck to increase development velocity 97. Build and Test Grids - Business 98. Build and Test Grids - Business A Unique Perspective Development Velocity is Revenue 99. Build and Test Grids - Business A Unique Perspective Development Velocity is Revenue Laundry Ops 100. Build and Test Grids - Business A Unique Perspective Development Velocity is Revenue Laundry Ops Now we talking disruption 101. Build and Test Grids - Adoption 102. Build and Test Grids - Adoption Next most common adoption path 103. Build and Test Grids - Adoption Next most common adoption path See as an efficient way to bring up many copies of a test environment efficiently 104. Build and Test Grids - Adoption Next most common adoption path See as an efficient way to bring up many copies of a test environment efficiently Surprisingly few producing a container from the build system 105. Build and Test Grids - Adoption Next most common adoption path See as an efficient way to bring up many copies of a test environment efficiently Surprisingly few producing a container from the build system The final mile 106. Build and Test Grids - Adoption Next most common adoption path See as an efficient way to bring up many copies of a test environment efficiently Surprisingly few producing a container from the build system The final mile Production adoption creating impedance 107. Build and Test Grids - Tools 108. Build and Test Grid - Tools Gap Build systems not container aware 109. Build and Test Grid - Tools Gap Build systems not container aware Build systems do not produce docker images 110. Build and Test Grid - Tools Gap Build systems not container aware Build systems do not produce docker images Build systems do not treat images as artifacts 111. Build and Test Grid - Tools Gap Build systems not container aware Build systems do not produce docker images Build systems do not treat images as artifacts Deployment systems are still, as a whole, immature 112. Build and Test Grid - Tools Gap Build systems not container aware Build systems do not produce docker images Build systems do not treat images as artifacts Deployment systems are still, as a whole, immature Private repos very immature 113. Build and Test Grids - Tools Available Jenkins - plugin Bamboo Docker Repository Quay.io 114. System Architecture 115. System Architecture - Defined Overloaded term 116. System Architecture - Defined Overloaded term Is concerned with how the various services of a software system interact 117. System Architecture - Defined Overloaded term Is concerned with how the various services of a software system interact Network, Data flow, request path, job management 118. System Architecture - Benefits 119. System Architecture - Benefits A separation of concerns leads to a âcode to the interfaceâ paradigm 120. System Architecture - Benefits A separation of concerns leads to a âcode to the interfaceâ paradigm Micro teamsâ micro-services can move at their own pace 121. System Architecture - Benefits A separation of concerns leads to a âcode to the interfaceâ paradigm Micro teamsâ micro-services can move at their own pace Only coordination between teams is on breaking changes. 122. System Architecture - Concerns 123. System Architecture - Concerns Very few coders out there who get it 124. System Architecture - Concerns Very few coders out there who get it Very few models for mere mortals to reason from 125. System Architecture - Business 126. System Architecture - Business Extraordinary increase in Development Team velocity 127. System Architecture - Business Extraordinary increase in Development Team velocity True competitive advantage 128. System Architecture - Business Extraordinary increase in Development Team velocity True competitive advantage Because of difficult in adoption, advantage will be lasting 129. System Architecture - Adoption 130. System Architecture - Adoption Micro service architecture is very rare in the wild (unicorns) 131. System Architecture - Adoption Micro service architecture is very rare in the wild (unicorns) Investment to move existing applications is high risk 132. System Architecture - Adoption Micro service architecture is very rare in the wild (unicorns) Investment to move existing applications is high risk Most shops are not mature/agile enough to realize the benefit 133. System Architecture - Tools 134. System Architecture - Tools Gap Meaningful materials on micro service architectures 135. System Architecture - Tools Gap Meaningful materials on micro service architectures Meaningful materials on async systems 136. System Architecture - Tools Available 12factor.net ? 137. Deployment 138. Deployment - Defined Docker Deployment promises A/B deployment 139. Deployment - Defined Docker Deployment promises A/B deployment Promises rolling release and rollback 140. Deployment - Benefits 141. Deployment - Benefits Easier to reason about deployment operations 142. Deployment - Benefits Easier to reason about deployment operations Configuration is not a concern, handled by development team 143. Deployment - Concerns 144. Deployment - Concerns Any discussion of rollback that involves a data store is still hand waving 145. Deployment - Concerns Any discussion of rollback that involves a data store is still hand waving Complexity: Different services need to be deployed in different ways 146. Deployment - Concerns Any discussion of rollback that involves a data store is still hand waving Complexity: Different services need to be deployed in different ways A/B deployment makes a number of assumptions about application architecture 147. Deployment - Concerns Any discussion of rollback that involves a data store is still hand waving Complexity: Different services need to be deployed in different ways A/B deployment makes a number of assumptions about application architecture No tools for the job 148. Deployment - Business 149. Deployment - Business Decreases deployment friction 150. Deployment - Business Decreases deployment friction Features get to production faster and more reliably 151. Deployment - Business Decreases deployment friction Features get to production faster and more reliably Significant, lasting competitive advantage 152. Deployment - Adoption 153. Deployment - Adoption Shops adopting CoreOS must adopt this some level of A/B deployment 154. Deployment - Adoption Shops adopting CoreOS must adopt this some level of A/B deployment Lack of tools is impeding adoption 155. Deployment - Tools 156. Deployment - Tools Gap A production ready container image has no place to go 157. Deployment - Tools Gap A production ready container image has no place to go Version aware scheduling - I have a new version of x, how do I deploy it based on policy y? 158. Deployment - Tools Available None yet Working on it StackEngine Tutum Fleet Dies Red Hat Google AWS 159. Food For Thought 160. Nourishment Black box production instrumentation - Care only about the container (tools donât exist) 161. Nourishment Black box production instrumentation - Care only about the container (tools donât exist) A/B Testing for Marketing 162. Nourishment Black box production instrumentation - Care only about the container (tools donât exist) A/B Testing for Marketing On Demand infrastructure (Pantheon) 163. Closing Thoughts 164. Closing Thoughts - Business 165. Business Developer adoption of Docker is only valuable as a first step. There is not enough benefit from it alone to justify the effort, it must inform system architecture and production operations (over time) 166. Business Developer adoption of Docker is only valuable as a first step. There is not enough benefit from it alone to justify the effort, it must inform system architecture and production operations (over time) Dockerâs system architecture ramifications have the potential to provide a significant and lasting competitive advantage 167. Business Developer adoption of Docker is only valuable as a first step. There is not enough benefit from it alone to justify the effort, it must inform system architecture and production operations (over time) Dockerâs system architecture ramifications have the potential to provide a significant and lasting competitive advantage Unlike most ops driven improvements derived from applying DevOps thinking, this must be developer driven since its greatest benefit is derived from system architecture 168. Business Developer adoption of Docker is only valuable as a first step. There is not enough benefit from it alone to justify the effort, it must inform system architecture and production operations (over time) Dockerâs system architecture ramifications have the potential to provide a significant and lasting competitive advantage Unlike most ops driven improvements derived from applying DevOps thinking, this must be developer driven since its greatest benefit is derived from system architecture The deployment model for Docker is promising, but still only done by unicorns (e.g. Netflix) 169. Closing Thoughts - DevOps 170. DevOps DevOps thought leaders are responsible for the holistic impact of technology decisions at the business level! 171. DevOps DevOps thought leaders are responsible for the holistic impact of technology decisions at the business level! DevOps thought leaders should be working with peers and collaborators in their company to determine if they can derive the proposed business benefits 172. DevOps DevOps thought leaders are responsible for the holistic impact of technology decisions at the business level! DevOps thought leaders should be working with peers and collaborators in their company to determine if they can derive the proposed business benefits Models must be developed that provide sensible direction for implementation (evolution not revolution) 173. DevOps DevOps thought leaders are responsible for the holistic impact of technology decisions at the business level! DevOps thought leaders should be working with peers and collaborators in their company to determine if they can derive the proposed business benefits Models must be developed that provide sensible direction for implementation (evolution not revolution) Tools are not there yet. Companies are showing up with the mission to address this, but it is very early days. 174. Should you be Considering Docker Adoption? 175. Thank You for Your Time and Comments. Boyd Hemphill @behemphi @stackengine https://www.friendbookmark.com/videos/983/infrastructure-deployment-with-docker-ansibleThis is an introduction to Docker & Ansible. It shows how Ansible can be used as orchestration too for Docker. There are 2 real world examples included with code examples in a Gist. Topics covered: 1. Infrastructure Deployment with 2. Who I am? â Robert Reiz â Software Developer â I started VersionEye â Software Dev since 1998 3. Agenda ❖ Intro to Docker ❖ Demo ❖ Intro to Ansible ❖ Demo 4. Shipment without Containers 5. 1956 Malcom McLean introduced the 40âContainer - ISO 668. > 15 Million inst. 2/3 of global trade run over 40âContainers! 6. The Logistic Problem 7. Same Problem in Software Dev. 8. Java ? ? ? Ruby ? ? ? Node.JS ? ? ? MySQL ? ? ? Dev-Env. Test-Env. Prod-Env. 9. Java JKD 1.8.14 - Win32 JKD 1.8.1 - Lnx-64 JDK 1.7-patch UNX Ruby 2.2.2 rvm 2.2.1 nat MRI 2.1.0 rubinius Node.JS 4.0 win 4.0 Linux 4.0 Linux MySQL 5.5 win 5.0 Linux 5.0 Linux Dev-Env. Test-Env. Prod-Env. 10. Java Ruby Node.JS MySQL Dev-Env. Test-Env. Prod-Env. 11. What is Docker? 12. What is Docker? ❖ Open Source Project started in March 2013 ❖ From the makers of dotCloud (PaaS). ❖ Received $162 Million Funding. ❖ Community grows rapidly! 13. What is Docker? ❖ Tiny VM (25 MB) ❖ Linux based - LXC Interface / libcontainer ❖ Own Namespaces and Cgroups! ❖ Shared resources with host system. ❖ Changes changed in Layers. Similar to Git! ❖ Originally not for Windows & Mac ! But â
 ->https://docs.docker.com/installation/windows/
 -> https://blog.docker.com/2016/03/docker-for-mac-windows-beta/ 14. Build - Ship - Run Docker-Hub Build RUN RUN RUN docker push docker pull Server Farm Production 15. Build 16. Dockerfile FROM ubuntu:14.10 MAINTAINER Robert Reiz ENV LANG en_US.UTF-8 RUN apt-get update RUN apt-get install -y --force-yes -q nginx ADD nginx.conf /etc/nginx/nginx.conf CMD nginx EXPOSE 80 17. Build - Dockerfile > docker build -t reiz/nginx:1.0.0 . docker image => reiz/nginx:1.0.0 18. Ship 19. Ship Docker Image > docker push reiz/nginx:1.0.0 20. Run 21. Fetch a Docker Image > docker pull reiz/nginx:1.0.0 Download docker image reiz/nginx:1.0.0 from Docker Hub to local Docker repository. 22. Run a Docker Container > docker run reiz/nginx:1.0.0 Creates a Docker container out of the Docker image reiz/nginx:1.0.0. It runs the nginx process. 23. More Commands > docker stop > docker start > docker top > docker logs > docker rm 24. Important ❖ A Docker Container doesnât store state! ❖ You can not ssh into a Docker Container! ❖ A container is supposed to run 1 process! 25. Shell 26. Get a Shell > docker run -it reiz/mongodb:3.2.0 /bin/bash Starts a new Docker container with an active shell. 27. Volumes 28. Mount a Volume > docker run -v/mnt/mongodb:/data -d reiz/mongodb:3.2.0 Mounts â/mnt/mongodbâ directory into the Docker container as â/dataâ. Keep the data on the host. Thatâs how you keep data persisted. 29. Environment Variables 30. Set environment variables > docker run --envLANG=en_US.UTF-8 -d reiz/mongodb:3.2.0 You can overwrite ENV variables from the Dockerfile here and also define completely new ones. 31. Links 32. Link Docker Containers > docker run âname mongodb -d versioneye/mongodb:1.0.2 > docker run âlinkmongodb:mongo versioneye/api:1.0.0 MONGO_PORT=tcp://172.1.10.1:27017 MONGO_PORT_27017_TCP=tcp://172.1.10.1:27017 MONGO_PORT_27017_TCP_ADDR=172.1.10.1 MONGO_PORT_27017_TCP_PORT=27017 MONGO_PORT_27017_TCP_PROTO=tcp Environmentvariablesareinjectedin2ndcontainer: 33. Link Docker Containers Linking only works on same hosts! 34. Docker Compose api: image: versioneye/rails_api:2.5.7 ports: - "9090:9090" container_name: "api" links: - mongodb:db - elasticsearch:es mongodb: image: reiz/mongodb:2.6.6_2 container_name: âmongodb" elasticsearch: image: reiz/elasticsearch:0.9.1 container_name: "elasticsearch" docker-compose.yml describes a whole set of Docker containers 35. Docker Compose > docker-compose up -d > docker-compose ps > docker-compose stop 36. Docker Compose > docker-compose build api > docker-compose up --no-deps -d api Updating a single container, not ALL of them. 37. Docker Compose > docker-compose scale worker=3 Scaling up containers 38. DEMO 39. Service Discovery 40. Service Discovery ❖ Environment Variables ❖ Mount configuration via volumes ❖ Linking ❖ Use a service like: etcd, zookeeper etcâ 41. Orchestration 42. Docker Orchestration ❖ CM-Tools (Chef, Puppet, Ansible, Salt) ❖ http://kubernetes.io/ ❖ https://coreos.com/ ❖ https://docs.docker.com/swarm/ ❖ https://www.openshift.com/ 43. 2012 VersionEye is running on Heroku. 44. 2013 VersionEye moves to AWS because of the Amazon Activate Program! 45. WWW API APP x 3 RabbitMQ Tasks MongoDB MongoDB MongoDB Elastic Search Memcached Crawlers x N VersionEye Infrastructure 46. Handcrafted Servers are ❖ hardtomaintain ❖ verytime/costintensive ❖ setupisnoteasilyreproducible ❖ manytimesverybuggy 47. Reasons for Ansible ❖ No Master ❖ No Agents ❖ Configuration in Yaml ❖ Very easy to learn 48. Server Server Server Server You SSH Ansible works via SSH. No Master Server! No Agent on the Server is required. 49. Installation 50. sudo pip install ansible 51. brew update brew install ansible 52. Ansible Concepts ❖ Inventory ❖ Playbooks ❖ Roles ❖ Tasks / Handlers / Vars ❖ Modules 53. Inventory 54. Inventory [mongo_master] 168.197.1.14 [mongo_slaves] 168.197.1.15 168.197.1.16 168.197.1.17 [www] 168.197.1.2 Inventory files are simple text files which describe your servers. IP Addresses or DNS Names grouped by names. 55. Inventory [mongo_master] 168.197.1.14 [mongo_slaves] mongo1.server mongo2.server mongo3.server [www] 168.197.1.2 List of target hosts. Usually located in /etc/ansible/hosts 56. Inventory [mongo_master] mongo-[a:c]-server [mongo_slaves] mongo[1:3].server [www] {{my_little_webserver}} Inventory files can take advantage of variables and enumerations 57. Playbooks 58. Simple Playbook --- - hosts: dev_servers user: ubuntu sudo: true roles: - java - memcached - hosts: www_servers user: ubuntu sudo: true roles: - java group name from the inventory file server auth Role which should be installed on the server 59. Roles / Modules 60. simpple role with apt module --- - name: update debian packages apt: update_cache=true - name: install Java JDK apt: name=openjdk-7-jdk state=present 61. apt module --- - name: update debian packages apt: update_cache=true - name: install Java JDK apt: name=openjdk-7-jdk state=present Documentation of this step! Module Parameters of the Module 62. --- - name: update debian packages apt: update_cache=true - name: upgrade packages apt: upgrade=full - name: ensure that basic packages are installed apt: name={{ item }} state=present with_items: - tree - wget - links2 - gcc - g++ - make - autoconf - automake - libssl-dev - libcurl4-openssl-dev 63. Thousands of Modules 64. shell module --- - name: do what you want shell: /opt/I_can_do_what_I_want.sh 65. A role directory 66. Variables --- app_dir: /var/www/versioneye tomcat/vars/main.yml --- - name: create versioneye directory command: mkdir -p {{ app_dir }} tomcat/tasks/main.yml 67. Handlers --- - name: restart mongodb service: name=mongod state=restarted mongo/handlers/main.yml - name: copy MongoDB configuration to the server copy: src=mongodb.conf dest=/etc/mongodb.conf notify: restart mongodb mongo/tasks/main.yml 68. Files mongo/files/mongodb.list - name: add MongoDB debian server to the list of servers copy: src=mongodb.list dest=/etc/apt/sources.list.d/mongodb.list mongo/tasks/main.yml 69. Real World Use case 70. Loadbalancer APP APP APP Seamless Web App Deployment @ VersionEye https://gist.github.com/reiz/238e70683bbfbc10bf4c 71. Loadbalancer APP APP APP Docker Hub 1. Build new Docker Image 2. push 3. run Seamless Web App Deployment @ VersionEye https://gist.github.com/reiz/238e70683bbfbc10bf4c 72. Loadbalancer APP APP APP Docker Hub 1. pull 2. run Seamless Web App Deployment @ VersionEye https://gist.github.com/reiz/238e70683bbfbc10bf4c 73. Loadbalancer APP APP APP Seamless Web App Deployment @ VersionEye Docker Hub 1. pull 2. run https://gist.github.com/reiz/238e70683bbfbc10bf4c 74. Loadbalancer APP APP APP Seamless Web App Deployment @ VersionEye https://gist.github.com/reiz/238e70683bbfbc10bf4c 75. Demo 76. ? ? ? @RobertReiz https://www.friendbookmark.com/videos/982/docker-and-containers-for-development-and-deploymentDocker is an Open Source engine to build, run, and manage containers. We'll explain what are Linux Containers, what powers them (under the hood), and what extra value Docker brings to the table. Then we'll see what the typical Docker workflow looks like from a developer point of view. We'll also give an Ops perspective, including deployment options. If you already saw a "Docker 101", consider this presentation as the February 2014 update! :-) Topics covered: 1. Best practices in development and deployment, with Docker and Containers February 2014âDocker 0.8.1 2. @jpetazzo ● Wrote dotCloud PAAS deployment tools âEC2, ● LXC, Puppet, Python, Shell, ÃMQ... Docker contributor âDocker-in-Docker, VPN-in-Docker, router-in-Docker... CONTAINERIZE ALL THE THINGS! ● Runs Docker in production âYou shouldn't do it, but here's how anyway! 3. Outline ● Why should I care? ● The container metaphor ● Very quick demo ● Working with Docker ● Building images ● Docker future 4. Outline ● Why should I care? ● The container metaphor ● Very quick demo ● Working with Docker ● Building images ● Docker future 5. Deploy everything ● webapps ● backends ● SQL, NoSQL ● big data ● message queues ● â and more 6. Deploy almost everywhere 7. Deploy almost everywhere YUP 8. Deploy almost everywhere YUP SOON 9. Deploy almost everywhere YUP SOON SOON 10. Deploy almost everywhere YUP SOON SOON 11. Deploy almost everywhere YUP SOON SOON CLI 12. Deploy almost everywhere YUP SOON SOON CLI 13. Deploy almost everywhere YUP SOON SOON CLI Yeah, right... 14. Deploy almost everywhere YUP SOON SOON CLI 15. Deploy almost everywhere ● Linux servers ● VMs or bare metal ● Any distro ● Kernel 3.8 (or RHEL 2.6.32) 16. Deploy reliably & consistently 17. Deploy reliably & consistently ● If it works locally, it will work on the server ● With exactly the same behavior ● Regardless of versions ● Regardless of distros ● Regardless of dependencies 18. Deploy efficiently ● Containers are lightweight â â ● Typical laptop runs 10-100 containers easily Typical server can run 100-1000 containers Containers can run at native speeds â Lies, damn lies, and other benchmarks: http://qiita.com/syoyo/items/bea48de8d7c6d8c73435 19. The performance! It's over 9000! 20. Outline ● Why should I care? ● The container metaphor ● Very quick demo ● Working with Docker ● Building images ● Docker future 21. â Container ? 22. High level approach: it's a lightweight VM ● own process space ● own network interface ● can run stuff as root ● can have its own /sbin/init (different from the host)  Machine Container  23. Low level approach: it's chroot on steroids ● can also not have its own /sbin/init ● container = isolated process(es) ● share kernel with host ● no device emulation (neither HVM nor PV)  Application Container  24. How does it work? Isolation with namespaces ● pid ● mnt ● net ● uts ● ipc ● user 25. pid namespace jpetazzo@tarrasque:~$ ps aux | wc -l 212 jpetazzo@tarrasque:~$ sudo docker run -t -i ubuntu bash root@ea319b8ac416:/# ps aux USER root root PID %CPU %MEM 1 0.0 0.0 16 0.0 0.0 (That's 2 processes) VSZ 18044 15276 RSS TTY 1956 ? 1136 ? STAT START S 02:54 R+ 02:55 TIME COMMAND 0:00 bash 0:00 ps aux 26. mnt namespace jpetazzo@tarrasque:~$ wc -l /proc/mounts 32 /proc/mounts root@ea319b8ac416:/# wc -l /proc/mounts 10 /proc/mounts 27. net namespace root@ea319b8ac416:/# ip addr 1: lo: mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 22: eth0: pfifo_fast state UP qlen 1000 mtu 1500 qdisc link/ether 2a:d1:4b:7e:bf:b5 brd ff:ff:ff:ff:ff:ff inet 10.1.1.3/24 brd 10.1.1.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::28d1:4bff:fe7e:bfb5/64 scope link valid_lft forever preferred_lft forever 28. uts namespace jpetazzo@tarrasque:~$ hostname tarrasque root@ea319b8ac416:/# hostname ea319b8ac416 29. ipc namespace jpetazzo@tarrasque:~$ ipcs ------ Shared Memory Segments -------key shmid owner perms 0x00000000 3178496 jpetazzo 600 0x00000000 557057 jpetazzo 777 0x00000000 3211266 jpetazzo 600 root@ea319b8ac416:/# ipcs ------ Shared Memory Segments -------key shmid owner perms ------ Semaphore Arrays -------key semid owner perms ------ Message Queues -------key msqid owner perms bytes 393216 2778672 393216 nattch 2 0 2 status dest bytes nattch status nsems used-bytes messages dest 30. user namespace ● ● ● No demo, but see LXC 1.0 (just released) UID 0â1999 in container C1 is mapped to UID 10000â11999 in host; UID 0â1999 in container C2 is mapped to UID 12000â13999 in host; etc. what will happen with copy-on-write? â double translation at VFS? â single root UID on read-only FS? 31. How does it work? Isolation with cgroups ● memory ● cpu ● blkio ● devices 32. memory cgroup ● keeps track pages used by each group: â file (read/write/mmap from block devices; swap) â anonymous (stack, heap, anonymous mmap) â active (recently accessed) â inactive (candidate for eviction) ● each page is  charged  to a group ● pages can be shared (e.g. if you use any COW FS) ● Individual (per-cgroup) limits and out-of-memory killer 33. cpu and cpuset cgroups ● keep track of user/system CPU time ● set relative weight per group ● pin groups to specific CPU(s) â Can be used to  reserve  CPUs for some apps â This is also relevant for big NUMA systems 34. blkio cgroups ● keep track IOs for each block device â read vs write; sync vs async ● set relative weights ● set throttle (limits) for each block device â read vs write; bytes/sec vs operations/sec Note: earlier versions ( /usr/local/etc/couchdb/local.d/docker.ini EXPOSE 8101 CMD ["/usr/local/bin/couchdb"] docker build -t jpetazzo/couchdb . 68. Authoring images with a Dockerfile ● Minimal learning curve ● Rebuilds are easy ● Caching system makes rebuilds faster ● Single file to define the whole environment! 69. Do you even Chef? Puppet? Ansible? Salt? 70. Docker and Puppet 71. Docker and Puppet ● Get a Delorean ● Warm up flux capacitors ● Time-travel to yesterday ● Check Brandon Burton's lightning talk ● Check my talk â Or â ● Get the slides, ask questions ☺ 72. Outline ● Why should I care? ● The container metaphor ● Very quick demo ● Working with Docker ● Building images ● Docker future 73. Coming Soon ● Network acceleration ● Container-specific metrics ● Consolidated logging ● Plugins (compute backends...) ● Orchestration hooks Those things are already possible, but will soon be part of the core. 74. Docker 1.0 ● Multi-arch, multi-OS ● Stable control API ● Stable plugin API ● Resiliency ● Signature ● Clustering 75. Recap Docker: ● Is easy to install ● Will run anything, anywhere ● Gives you repeatable builds ● Enables better CI/CD workflows ● Is backed by a strong community ● Will change how we build and ship software 76. Thank you! Questions? http://docker.io/ http://docker.com/ @docker @jpetazzo https://www.friendbookmark.com/videos/981/docker-devops-dockers-impact-on-traditional-devops-toolchainA talk given to Container Summit 2015, Taipei on 2015-12-10. 【Docker 對傳統 DevOps 工具鏈的衝擊】 Docker 爆紅，以及隨之而來的生態圈，衝擊傳統 DevOps 工具鏈，任何人都無法忽視 Docker 帶來的威脅或機會。 已經上手、或正在評估 DevOps 工具的我們，該如何看待這個新局面？對哪些舊方法該持保留態度？對哪些新方法該積極學習？ 本講座會鳥瞰檯面上的 DevOps 工具，提出新舊交替世代的建議方針。 Topics covered: 1. William Yeh Architect @ Gogolook- 2015-12-10 2. Albert Camus 3. - 4. 1. How to recreate your system 2. How to safely change your system 3. When something has gone wrong Do you have basic infrastructure? (2015-11-18) http://www.robustperception.io/do-you-have-basic-infrastructure/ 5. public network 1. How to recreate your system 2. How to safely change your system 3. When something has gone wrong private network CDN LB API servers DB servers 1.1 - 1.2 - 6. 1. How to recreate your system 2. How to safely change your system 3. When something has gone wrong private network CDN LB API servers DB servers public network .1 - .2 - 7. 1. How to recreate your system 2. How to safely change your system 3. When something has gone wrong private network CDN LB API servers DB servers public network detection recovery diagnosis 8. public network 1. How to recreate your system 2. How to safely change your system 3. When something has gone wrong private network CDN LB API servers DB servers 1.1 - 1.2 - 9. hardware bare metal OS runtime app VM hardware OS runtime app hypervisor 10. VM hardware OS runtime app hypervisor 11. VM hardware OS runtime app hypervisor Docker hardware OS runtime app hypervisor container 12. Docker hardware OS runtime app hypervisor container We know: â Container is faster than VM â Container is lightweight â Dockerized app anywhereâ What ifâ â An OS runs only containers? 13. Docker hardware OS runtime app hypervisor container CoreOS RancherOS Red Hat Atomic VMware Photon Snappy Ubuntu Core Windows Nano Server 14. Docker hardware OS runtime app hypervisor container Container per VM hardware OS runtime app hypervisor container 15. Container per VM hardware OS runtime app hypervisor container Intel Clear Linux http://www.ithome.com.tw/news/96119 Hyper https://hyper.sh/ 16. Container per VM hardware OS runtime app hypervisor container Unikernel hardware library OS app hypervisor unikernel
 app stack 17. Unikernel hardware library OS app hypervisor unikernel
 app stack Unikernels are constructed by using âlibrary operating systems,â from which the developer selects only the minimal set of services required for an application to run. 18. Unikernel hardware library OS app hypervisor unikernel
 app stack MirageOS https://mirage.io/ Boxfuse https://boxfuse.com/ ClickOS Clive HaLVM LING Rump Kernels OSv 19. Unikernel image Immutable infrastructure 20. public network 1. How to recreate your system 2. How to safely change your system 3. When something has gone wrong private network CDN ELB API servers DB servers 1.1 - 1.2 - 21. service consolidation resourceisolation native app VM container per VM unikernel Docker 22. public network 1. How to recreate your system 2. How to safely change your system 3. When something has gone wrong private network CDN LB API servers DB servers 1.1 - 1.2 - 23. public network private network API servers JDK app server (Tomcat, â) app (jar, war, â) Node.js runtime npm app source Python runtime pip uWSGI/Gunicorn app source 24. public network private network DB servers JDK Elasticsearch JDK CassandraMongoDBMySQL dependenciesdependencies 25. public network private network LB HAProxynginx AWS ELB dependenciesdependencies 26. JDK app server (Tomcat, â) app (jar, war, â) Node.js runtime npm app source Python runtime pip uWSGI/Gunicorn app source JDK Elasticsearch JDK CassandraMongoDBMySQL dependenciesdependencies HAProxynginx AWS ELB dependenciesdependencies 27. JDK app server (Tomcat, â) app (jar, war, â) Node.js runtime npm app source Python runtime pip uWSGI/Gunicorn app source JDK Elasticsearch JDK CassandraMongoDBMySQL dependenciesdependencies HAProxynginx AWS ELB dependenciesdependencies DevOps tools â Configuration management â Build system â Deployment pipeline â Continuous integration 28. JDK app server (Tomcat, â) app (jar, war, â) Node.js runtime npm app source Python runtime pip uWSGI/Gunicorn app source JDK Elasticsearch JDK CassandraMongoDBMySQL dependenciesdependencies HAProxynginx AWS ELB dependenciesdependencies Docker runtime app image immutable image versioned image dev/prod parity Dockerfile docker build docker push docker pull 29. public network 1. How to recreate your system 2. How to safely change your system 3. When something has gone wrong private network CDN ELB API servers DB servers 1.1 - 1.2 - 30. uniform Docker Config management will only be used to install Docker, an orchestration system, configure PAM/SSH auth, and tune OS sysctl values. traditional DevOps toolchain tedious â Basically anything not having to do with app deployment. https://blog.containership.io/containers-vs-config-management-e64cbb744a94 31. public network 1. How to recreate your system 2. How to safely change your system 3. When something has gone wrong private network CDN LB API servers DB servers 1.1 - 1.2 - 32. public network private network API servers As the number of machines growsâ how to ensure better allocation? 33. pets cattle 34. pets cattle naming? 35. pets cattle dispensable? 36. pets cattle naming? dispensable? 37. pets cattle PaaS Hadoop MapReduce AWS Lambda AWS Kinesis Google Dataflow 38. public network private network API servers As the number of machines growsâ how to ensure better allocation? 39. Traditional app OS runtime app tight interaction host resources pets 40. https://prezi.com/e7sdy9rdujgp 41. public network private network API servers better mobility cattle 42. public network private network API servers even better mobilitycontainer cattle independent of underlying machines 43. cattle Docker Swarm Mesos Kubernetes allocation, orchestration 44. public network 1. How to recreate your system 2. How to safely change your system 3. When something has gone wrong private network CDN ELB API servers DB servers 1.1 - 1.2 - 45. Docker Swarm Mesos Kubernetes Config management will only be used to install Docker, an orchestration systemâ traditional DevOps toolchain â Donât assume too much about underlying infrastructure. cattlepets 46. cattle Long Running Services â Aurora â Marathon â Singularity â SSSP Batch Scheduling â Chronos â Jenkins â JobServer Big Data Processing â Cray Chapel â Dpark â Exelixi â Hadoop â Hama â MPI â Spark â Storm Data Storage â Cassandra â Elasticsearch â Hypertable Mesos framework 47. cattle Mesos 48. 1. How to recreate your system 2. How to safely change your system 3. When something has gone wrong .1 - .2 - private network CDN LB API servers DB servers public network 49. private network LB API servers DB servers â Rolling upgrade â Blue/green deployment â Canary deployment Traditional app â in-place update â immutable infra 50. public network private network API servers immutable imagescontainer cattle independent of underlying machines 51. cattle Docker Swarm Mesos Kubernetes allocation, orchestration 52. kubectl rolling-update my-nginx --image=nginx:1.9.1 â Rolling upgrade â Canary deployment â use label Kubernetes 53. Mesos http://blog.qubit.com/opensourcing-bamboo-automated-mesos-marathon-load-balancing http://www.slideshare.net/johnadowns/making-developers-happier-with-mesos-docker-and-marathon 54. Universal Control Plane https://www.docker.com/universal-control-plane 55. public network 1. How to recreate your system 2. How to safely change your system 3. When something has gone wrong private network CDN ELB API servers DB servers .1 - .2 - 56. Docker Swarm Mesos Kubernetes traditional DevOps toolchain Consider the benefits: â immutable infrastructure â automated allocation â automated orchestration cattlepets 57. private network CDN LB API servers DB servers Key Takeaways 58. 1. How to recreate your system 2. How to safely change your system 3. When something has gone wrong Do you have basic infrastructure? (2015-11-18) http://www.robustperception.io/do-you-have-basic-infrastructure/ 59. service consolidation resourceisolation native app VM container per VM unikernel Docker Immutable infrastructure 60. uniform Docker Config management will only be used to install Docker, an orchestration systemâ anything other than app deployment. traditional DevOps toolchain tedious 61. Docker Swarm Mesos Kubernetes traditional DevOps toolchain â Donât assume too much about underlying infrastructure. cattlepets 62. cattle Docker Swarm Mesos Kubernetes allocation, orchestration 63. http://send.wtf/docker2015 https://www.friendbookmark.com/videos/980/docker-by-example-basicsDocker has created enormous buzz in the last few years. Docker is a open-source software containerization platform. It provides an ability to package software into standardised units on Docker for software development. In this hands-on introductory session, I introduce the concept of containers, provide an overview of Docker, and take the participants through the steps for installing Docker. The main session involves using Docker CLI (Command Line Interface) - all the concepts such as images, managing containers, and getting useful work done is illustrated step-by-step by running commands. Topics covered: 1. Lorem Ipsum Dolor Docker by Example Ganesh & Hari ganesh@codeops.tech hari@codeops.tech Using a Visual Approach 2. Why Docker? 3. Why Docker? 4. What is Docker? Docker is an open-source project that automates the deployment of applications inside software containers. 5. What is Docker? âan open platform for developers and sysadmins to build, ship, and run distributed applicationsâ 6. What is Docker? Source: https://en.wikipedia.org/wiki/Docker_(software) 7. Pop quiz Docker is written in: A. Java language B. C language C. D language D. Go language 8. Pop quiz: answer Docker is written in Go language Good read: Why Did We Decide to Write Docker in Go? 9. VMs vs. Docker 10. VMs vs. Docker 11. VMs vs. containers Simulates a physical machine Provides a local file system Can be accessed over a network Full and independent guest operating system Virtualized device drivers Strong resource and memory management Huge memory foot-print Needs a hypervisor 12. Docker accesses virtualisation features of Linux 13. Native Docker support on Windows Source: https://i2.wp.com/blog.docker.com/wp-content/uploads/windows.png?w=975&ssl=1 https://i2.wp.com/blog.docker.com/wp-content/uploads/windows.png?w=975&ssl=1 14. Docker for DevOps 15. Docker becoming popular over time Google Trends: https://www.google.co.uk/trends/explore?q=%2Fm%2F0wkcjgj&hl=en-US 16. Whatâs covered in this session? 17. Getting Started 18. Essential Docker components 19. Installing Docker Install it from âhttps://www.docker.com/products/overview" 20. Official Docker images? From âhttps://hub.docker.com/explore/" 21. Finding Docker version 22. How to find my Docker version? $ docker -v Docker version 1.12.0-rc4, build e4a0dbc, experimental 23. Finding details of a Docker installation 24. Can I install Docker from commandline? Yes! from get.docker.com # This script is meant for quick & easy install via: # 'curl -sSL https://get.docker.com/ | sh' # or: # 'wget -qO- https://get.docker.com/ | sh' 25. How to do âhello worldâ in Docker? $ docker run docker/whalesay cowsay Hello world 26. How to do âhello worldâ in Docker? $ docker run docker/whalesay cowsay "Hello world" Runs a command in a new container Base image for creating the container Command name to run within the container Argument to the âcowsayâ command 27. How to do âhello worldâ in Docker? $ docker run -it hello-world $ docker run -it hello-world Hello from Docker! This message shows that your installation appears to be working correctly. To generate this message, Docker took the following steps: 1. The Docker client contacted the Docker daemon. 2. The Docker daemon pulled the "hello-world" image from the Docker Hub. 3. The Docker daemon created a new container from that image which runs the executable that produces the output you are currently reading. 4. The Docker daemon streamed that output to the Docker client, which sent it to your terminal. To try something more ambitious, you can run an Ubuntu container with: $ docker run -it ubuntu bash Share images, automate workflows, and more with a free Docker Hub account: https://hub.docker.com For more examples and ideas, visit: https://docs.docker.com/engine/userguide/ 28. How to get help on commands to use? Use âdocker -hâ command, as in: $ docker -h Usage: docker [OPTIONS] COMMAND [arg...] docker [ --help | -v | --version ] A self-sufficient runtime for containers. Options: --config=~/.docker Location of client config files -D, --debug Enable debug mode -H, --host=[] Daemon socket(s) to connect to -h, --help Print usage -l, --log-level=info Set the logging level â Commands: attach Attach to a running container 29. Docker commands look like Linux commands - so familiarity with Linux commands can really help to get up to speed quickly with Docker. 30. Docker Images 31. How to get list of images? 32. How to search for an image? 33. How to get an image? Use âdocker pull â command In my case debian image was already pulled. If it were not there, Docker would have pulled it afresh 34. Choose smaller images ❖ Example: Choose Alpine vs. Fedora (5 MB vs. 205 MB) alpine latest 4e38e38c8ce0 4 weeks ago 4.799 MB fedora latest f9873d530588 4 weeks ago 204.4 MB ❖ Prefer choosing a smaller base image that provides equivalent functionality (for your requirement) instead of choosing a larger one 35. How to get details of an image? Use âdocker inspect â command docker inspect debian [ { "Id": "sha256:1b088884749bd93867ddb48ff404d4bbff09a17af8d95bc863efa5d133f87b78", "RepoTags": [ "debian:latest" ], "RepoDigests": [ "debian@sha256:8b1fc3a7a55c42e3445155b2f8f40c55de5f8bc8012992b26b570530c4bded9e" ], "Parent": "", "Comment": "", "Created": "2016-06-09T21:28:43.776404816Z", "Container": "2f3dcd897cf758418389d50784c73b43b1fd7db09a80826329496f05eef7b377", "ContainerConfig": { "Hostname": "6250540837a8", "Domainname": "", "User": "", "AttachStdin": false, "AttachStdout": false, "AttachStderr": false, "Tty": false, "OpenStdin": false, // ... 36. How to see âlayersâ in an image? Use âdocker history â command Each of these lines are layers and the size column shows the exact size of each layer in the image 37. Use tools to visualise layers ❖ You can use the online tool imagelayers.io to visualise the layers of an image 38. How can I load and store images? Use âdocker saveâ and âdocker loadâ commands 39. How do I delete an image? Use âdocker rmi â $ docker images alpine REPOSITORY TAG IMAGE ID CREATED SIZE alpine latest 4e38e38c8ce0 4 weeks ago 4.799 MB $ docker rmi alpine Untagged: alpine:latest Untagged: alpine@sha256:3dcdb92d7432d56604d4545cbd324b14e647b313626d99b889d0626de158f73a $ docker images alpine REPOSITORY TAG IMAGE ID CREATED SIZE 40. How to delete all docker images? Use â$docker rmi $(docker images -q)â docker images -q lists all image ids 41. Avoid âimage sprawlâ // output from the docker bench security tool [INFO] 6.4 - Avoid image sprawl [INFO] * There are currently: 60 images ❖ Remove unused images and release disk space 42. How to find âdangling imagesâ? Use âdocker images -f "dangling=true"â $ docker images -f "dangling=true" REPOSITORY TAG IMAGE ID CREATED SIZE 777f9424d24d 7 minutes ago 125.2 MB 3d02168f00fc 12 days ago 34.22 MB 0f192147631d 3 weeks ago 132.8 MB 43. Remove âdangling imagesâ ❖ Remove âdangling imagesâ using the command â$docker rmi $(docker images -f "dangling=true" -q)â 44. Using Registry & Repository 45. How to push my image to Docker Hub? $ docker tag myjavaapp gsamarthyam/myfirstjavaprog:latest $ docker push gsamarthyam/myfirstjavaprog:latest The push refers to a repository [docker.io/gsamarthyam/myfirstjavaprog] a97e2e0314bc: Pushed 3b9964bc9417: Pushed de174b528b56: Pushed // elided the output latest: digest: sha256:1618981552efb12afa4e348b9c0e6d28f0ac4496979ad0c0a821b43547e13c13 size: 2414 $ 46. How to pull my image from Docker Hub? $ docker pull gsamarthyam/myfirstjavaprog:latest latest: Pulling from gsamarthyam/myfirstjavaprog Digest: sha256:1618981552efb12afa4e348b9c0e6d28f0ac4496979ad0c0a821b43547e13c13 // output elided â $ docker images REPOSITORY TAG IMAGE ID CREATED SIZE myjavaapp latest 0d7a3a12ba9d About an hour ago 669.2 MB gsamarthyam/myfirstjavaprog latest 0d7a3a12ba9d About an hour ago 669.2 MB // output elided â $ docker run gsamarthyam/myfirstjavaprog hello world $ 47. How do I create and use my own registry? $ docker pull registry Using default tag: latest latest: Pulling from library/registry e110a4a17941: Already exists 2ee5ed28ffa7: Pull complete d1562c23a8aa: Pull complete 06ba8e23299f: Pull complete 802d2a9c64e8: Pull complete Digest: sha256:1b68f0d54837c356e353efb04472bc0c9a60ae1c8178c9ce076b01d2930bcc5d Status: Downloaded newer image for registry:latest $ docker run -d -p5000:5000 registry 7768bed98a5e1916a820c84906e1f21cfc84888a934c140ad22e19cee5e2541d Pull the âregistryâ image and run the container You can now push/pull images from this private registry 48. Docker Containers 49. How to get list of containers? 50. How to run a container? Use âdocker run OPTIONS CMD ARGSâ docker run fedora /bin/echo 'Hello world' Command name Image name Command argument $ docker run fedora /bin/echo 'Hello world' Hello world $ 51. How to run a container interactively? $ docker run -t -i fedora /bin/bash [root@00eef5289c91 /]# pwd / [root@00eef5289c91 /]# whoami root [root@00eef5289c91 /]# ls bin boot dev etc home lib lib64 lost+found media mnt opt proc root run sbin srv sys tmp usr var [root@00eef5289c91 /]# cc bash: cc: command not found [root@00eef5289c91 /]# gcc bash: gcc: command not found [root@00eef5289c91 /]# java bash: java: command not found [root@00eef5289c91 /]# tar bash: tar: command not found [root@00eef5289c91 /]# exit exit $ docker run -t -i fedora /bin/bash Create a terminal to interact with short for ââinteractive" 52. Cobbâs totem - the top 53. Running a container - Totem? 54. Running a container - Totem? $ hostname ganesh $ docker run -it alpine /bin/sh / # hostname b4ebae46b156 / # ps -a PID USER TIME COMMAND 1 root 0:00 /bin/sh 6 root 0:00 ps -a / # exit $ ps -a PID TTY TIME CMD 15327 ttys001 0:00.02 login -pf gsamarthyam 15328 ttys001 0:00.27 -bash 55. How to run a container in the background? $ docker run -d ubuntu /bin/sh -c "while true; do echo current date and time is: $(date); sleep 10; done" 9128bf57e03c3b32f0bf784a92332953996236d7e358a77c62c10bdec95fd5b9 $ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 9128bf57e03c ubuntu "/bin/sh -c 'while tr" About a minute ago Up About a minute lonely_einstein $ docker logs 9128bf57e03c3b32f0bf784a92332953996236d7e358a77c62c10bdec95fd5b9 current date and time is: Fri Jul 22 15:42:49 IST 2016 current date and time is: Fri Jul 22 15:42:49 IST 2016 current date and time is: Fri Jul 22 15:42:49 IST 2016 current date and time is: Fri Jul 22 15:42:49 IST 2016 // output elided short for ââdetachâ and it runs container in the background 56. How to expose a port? $ docker run -d -p 80:80 nginx 9128bf57e03c3b32f0bf784a92332953996236d7e358a77c62c10bdec95fd5b9 host port (machine on which this command is run) mapped port - nginx "PortBindings": { "80/tcp": [ { "HostIp": "", "HostPort": "80" } ] }, $ docker inspect 9128bf57e03c3b32f0bf784a92332953996236d7e358a77c62c10bdec95fd5b9 57. Using Nginx Type http://localhost:80 in your browser window 58. How to expose a port? $ docker run -d -p 80 --name minenginx nginx 9128bf57e03c3b32f0bf784a92332953996236d7e358a77c62c10bdec95fd5b9 host port (machine on which this command is run); since no explicit mapped port number is provided, a random port number is assigned $ docker inspect minenginx "Ports": { "443/tcp": null, "80/tcp": [ { "HostIp": "0.0.0.0", "HostPort": "32770" } ] }, randomly assigned and mapped port number (by docker) 59. How to expose all exposed ports? $ docker run -d -P --name minenginx nginx 6b873116f198f4235e3eee1b2085e0312eaa0067217da614c62e2ce55a8c8d4e $ docker port minenginx 443/tcp -> 0.0.0.0:32771 80/tcp -> 0.0.0.0:32772 -P publishes all exposed ports to random ports; in this case, ports 443 and 80 are respectively mapped to 32771 and 32772 60. How to attach to a running container? $ docker run -d ubuntu /bin/sh -c "while true; do echo current date and time is: $ (date); sleep 10; done" acc349675098a0133366076f2082db6171ee4a0cd2e1e45ada9a485684ea4c01 $ docker attach acc349675098a0133366076f2082db6171ee4a0cd2e1e45ada9a485684ea4c01 current date and time is: Mon Aug 1 10:30:13 IST 2016 current date and time is: Mon Aug 1 10:30:13 IST 2016 short for ââdetachâ and it runs container in the background The âattachâ command attaches to a running container 61. How to detach from a running container (without exiting)? From docker documentation # To detach the tty without exiting the shell, # use the escape sequence Ctrl-p + Ctrl-q # note: This will continue to exist in a stopped state once exited (see "docker ps -a") $ docker run -it alpine /bin/sh / # echo "hello" hello / # $ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 99d946d9b4e0 alpine "/bin/sh" 15 seconds ago Up 14 seconds gloomy_mcclintock $ 62. How do I create an image from a running container? Use âdocker commitâ command $ docker run -d alpine echo "hello world" 9884347880f62f7c5d43702c3d701e3b87a49f9bdde5843380af1479f4dc0755 $ docker logs 9884347880f62f7c5d43702c3d701e3b87a49f9bdde5843380af1479f4dc0755 hello world $ docker commit -m "my first image from container" 9884347880f62f7c5d43702c3d701e3b87a49f9bdde5843380af1479f4dc0755 myalpine:latest sha256:b707ef35394c365bece70240213942e43da7f882107d30482ad6bec6b4bacfb7 $ docker images REPOSITORY TAG IMAGE ID CREATED SIZE myalpine latest b707ef35394c 18 hours ago 4.799 MB $ docker run -it b707ef35394c365bece70240213942e43da7f882107d30482ad6bec6b4bacfb7 hello world $ 63. Avoid âdocker commitâ ❖ Avoid creating docker images manually (e.g., using âdocker commitâ); rather automate the image build process (using Dockerfile and âdocker buildâ) 64. How to get list of containers? Use âdocker psâ command $ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 3651758ff308 wordpress:latest "/entrypoint.sh apach" 2 days ago Up 2 days 0.0.0.0:8000->80/tcp mywordpress_wordpress_1 b95388054539 mysql:5.7 "docker-entrypoint.sh" 2 days ago Up 2 days 3306/tcp mywordpress_db_1 65. How do I see all the containers? Use âdocker ps -aâ command $ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 2c378c6b84b1 fedora "/bin/echo 'Hello wor" 4 minutes ago Exited (0) 4 minutes ago grave_thompson c4b2db95f268 hello-world "/hello" 5 minutes ago Exited (0) 5 minutes ago amazing_jones 2dcd9d0caf6f 777f9424d24d "/bin/bash" 42 minutes ago Exited (0) 42 minutes ago prickly_khorana 3651758ff308 wordpress:latest "/entrypoint.sh apach" 2 days ago Up 2 days 0.0.0.0:8000->80/tcp mywordpress_wordpress_1 b95388054539 mysql:5.7 "docker-entrypoint.sh" 2 days ago Up 2 days 3306/tcp mywordpress_db_1 4b984664f9aa golang:latest "go run myapp.go" 2 days ago Exited (1) 2 days ago mydocker_app_1 63cd7661a8ad hello-world "/hello" 2 days ago Exited (0) 2 days ago adoring_sammet c191fbeae884 ubuntu "/bin/bash" 2 days ago Exited (0) 2 days ago clever_mcclintock 08e173332d46 docker/whalesay "cowsay Hello world" 2 days ago Exited (0) 2 days ago tender_joliot 6322b8204a5d 0f192147631d "/bin/bash" 9 days ago Exited (0) 9 days ago desperate_aryabhata ... 66. Explicitly remove exited containers ❖ Explicitly use --rm to remove the container from the file system - otherwise, even if the container exits, it is not cleaned up yet (and will hog memory). 67. How do I remove a container? Use âdocker rmâ command $ docker stop mywordpress_db_1 mywordpress_db_1 $ docker rm mywordpress_db_1 mywordpress_db_1 You have to first stop a container before trying to remove it 68. How do I remove all the containers? Use âdocker stop $(docker ps -a -q)â and âdocker rm $(docker ps -a -q)â commands $ docker stop $(docker ps -a -q) 00eef5289c91 8553eebfab94 696a04db91db // rest of the output elided $ docker rm $(docker ps -a -q) 00eef5289c91 8553eebfab94 696a04db91db // rest of the output elided $ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Note how the output shows no containers 69. State transition 70. Using nginx $ docker run --name mynginx -P -d nginx 561e15ac1848cf481f89bb161c23dd644f176b8f142fe617947e06f095e0953f $ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 561e15ac1848 nginx "nginx -g 'daemon off" 18 hours ago Up About a minute 0.0.0.0:32771->80/tcp, 0.0.0.0:32770->443/tcp mynginx $ curl localhost:32771 Welcome to nginx! body { width: 35em; margin: 0 auto; font-family: Tahoma, Verdana, Arial, sans-serif; } // rest of the output elided ... Nginx exposes ports 80 and 443; -P maps them randomly in the ports range 49153 and 65535 71. Using nginx $ cat Dockerfile FROM nginx:latest MAINTAINER Ganesh Samarthyam ADD ./index.html /usr/share/nginx/html/index.html EXPOSE 80 $ cat index.html welcome to Dockerizing apps! $ docker build . Sending build context to Docker daemon 3.072 kB // output elided ... Removing intermediate container b043a75a4e1c Successfully built 1aae04309f8b $ docker images REPOSITORY TAG IMAGE ID CREATED SIZE 1aae04309f8b 6 seconds ago 182.8 MB $ docker run -p 80:80 -d 1aae04309f8b 984c179231188445289e70d854250e4e981b77a899208360db4466e73930be42 $ curl localhost:80 welcome to Dockerizing apps! $ Type âlocalhost:80â in the browser address bar 72. How do I run a C program? $ docker pull gcc Using default tag: latest latest: Pulling from library/gcc 5c90d4a2d1a8: Already exists ab30c63719b1: Already exists c6072700a242: Already exists abb742d515b4: Already exists d32a4c04e369: Pull complete 276c31cf0a4c: Pull complete a455d29f9189: Pull complete dcfe5869552b: Pull complete Digest: sha256:35256b5f4e4d5643c9631c92e3505154cd2ea666d2f83812b418cfdb1d5866e8 Status: Downloaded newer image for gcc:latest $ 73. How do I run a C program? $ docker pull ubuntu:latest latest: Pulling from library/ubuntu 43db9dbdcb30: Pull complete 85a9cd1fcca2: Pull complete c23af8496102: Pull complete e88c36ca55d8: Pull complete Digest: sha256:7ce82491d6e35d3aa7458a56e470a821baecee651fba76957111402591d20fc1 Status: Downloaded newer image for ubuntu:latest $ docker run -i -t ubuntu /bin/bash root@c191fbeae884:/# gcc bash: gcc: command not found root@c191fbeae884:/# apt-get update // elided the output root@c191fbeae884:/# apt-get install gcc // elided the output root@c191fbeae884:/# cat > hello.c int main() { printf("hello worldnâ); } root@c191fbeae884:/# gcc -w hello.c root@c191fbeae884:/# ./a.out hello world root@c191fbeae884:/# 74. How do I run a C program? $ cat Dockerfile FROM gcc:latest MAINTAINER Ganesh Samarthyam version: 0.1 COPY . /usr/src/mycapp WORKDIR /usr/src/mycapp RUN gcc -o first first.c CMD ["./first"] $ cat first.c #include int main() { printf("hello worldn"); } $ docker build . -t"mycapp:latest" Sending build context to Docker daemon 3.072 kB Step 1 : FROM gcc:latest ---> a0b516dc1799 // .. steps elided ... Step 6 : CMD ./first ---> Using cache ---> f99e7f18fa42 Successfully built f99e7f18fa42 $ docker run -it mycapp hello world 75. How do I run a Java program? $ cat Dockerfile FROM java:latest COPY . /usr/src/ WORKDIR /usr/src/ RUN javac hello.java CMD ["java", "hello"] $ cat hello.java class hello { public static void main(String []args) { System.out.println("hello world"); } } $ docker build . -t"myjavaapp:latest" Sending build context to Docker daemon 3.072 kB Step 1 : FROM java:latest ---> 264282a59a95 // intermediate steps elided Successfully built 0d7a3a12ba9d $ docker images REPOSITORY TAG IMAGE ID CREATED SIZE myjavaapp latest 0d7a3a12ba9d About an hour ago 669.2 MB $ docker images REPOSITORY TAG IMAGE ID CREATED SIZE myjavaapp latest 0d7a3a12ba9d About an hour ago 669.2 MB 7cfb4bdf47a7 About an hour ago 669.2 MB // rest of the output elided $ docker run myjavaapp hello world 76. Beware of âcontainer sprawlâ ❖ Application broken to run in âtoo many containersâcan be difficult to deal with! âBreaking deployments into more functional discrete parts is smart, but that means we have MORE PARTS to manage. There's an inflection point between separation of concerns and sprawl.â -- Rob Hirschfeld (CEO of RackN and OpenStack Foundation board member) 77. Pop quiz What happens when you execute this on the command- line? docker run debian /bin/sh A. A prompt from the shell of created container will be thrown to you B. A container is created and then exited immediately C. A container is created and executes in the detached mode; you can attach to it later using the container id D. Docker CLI issues the error: Error response from daemon: No command specified. 78. Pop quiz: answer When you execute this command, docker run debian /bin/sh A container is created and then exited immediately. $ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 4c12998fd392 debian "/bin/bash" 6 seconds ago Exited (0) 5 seconds ago sick_panini 79. Pop quiz What happens when you execute this on the command- line? docker run -itd debian A. You get âError response from daemon: No command specified.â B. The created container runs in the detached mode C. You get âunknown shorthand flag: -itdâ D. A shell from the created container is returned to you 80. Pop quiz: answer When you execute docker run -itd debian the created container runs in the detached mode! $ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES a53779a74904 debian "/bin/bash" 2 minutes ago Up 2 minutes agitated_aryabhata 81. Building images using Dockerfile 82. Different ways to create images docker commit Build an image from a container docker build Create an image from a Dockerfile by executing the build steps given in the file docker import Create a base image by importing from a tarball. [import is mainly used for creating base-images; first two options are widely used] 83. Dockerfile - key instructions FROM The base image for building the new docker image; provide âFROM scratchâ if it is a base image itself MAINTAINER The author of the Dockerfile and the email RUN Any OS command to build the image CMD Specify the command to be started when the container is run; can be overridden by the explicit argument when providing docker run command ADD Copies files or directories from the host to the container in the given path EXPOSE Exposes the specified port to the host machine 84. How can I create an image from a Dockerfile? Use docker build command âDockerfileâ - its like Makefile for Docker $ cat myimage/Dockerfile FROM ubuntu RUN echo "my first image" > /tmp/first.txt $ docker build myimage Sending build context to Docker daemon 2.048 kB Step 1 : FROM ubuntu ---> ac526a356ca4 Step 2 : RUN echo "my first image" > /tmp/first.txt ---> Running in 18f62f47d2c8 ---> 777f9424d24d Removing intermediate container 18f62f47d2c8 Successfully built 777f9424d24d $ docker images | grep 777f9424d24d 777f9424d24d 4 minutes ago 125.2 MB $ docker run -it 777f9424d24d root@2dcd9d0caf6f:/# ls bin boot core dev etc home lib lib64 media mnt opt proc root run sbin srv sys tmp usr var root@2dcd9d0caf6f:/# cat /tmp/first.txt my first image root@2dcd9d0caf6f:/# exit exit $ 85. How to name/tag an image when building? Use âdocker build -t"imagename:tag"â command $ docker build myimage -t"myfirstimage:latest" Sending build context to Docker daemon 2.048 kB Step 1 : FROM ubuntu ---> ac526a356ca4 Step 2 : RUN echo "my first image" > /tmp/first.txt ---> Using cache ---> 777f9424d24d Successfully built 777f9424d24d $ docker images myfirstimage REPOSITORY TAG IMAGE ID CREATED SIZE myfirstimage latest 777f9424d24d 58 minutes ago 125.2 MB $ 86. Dockerfile for running a Java program $ cat HiHello.java import java.io.*; import java.net.InetSocketAddress; import com.sun.net.httpserver.*; public class HiHello { public static void main(String[] args) throws Exception { HttpServer server = HttpServer.create(new InetSocketAddress(8080), 0); server.createContext("/hi", (HttpExchange t) -> { try { String response = "hellon"; t.sendResponseHeaders(200, response.length()); try(OutputStream os = t.getResponseBody()) { os.write(response.getBytes()); } } catch (IOException ioe) { System.err.println("Error writing to outputstream: " + ioe); System.exit(-1); } } ); server.start(); } } $ curl localhost:8080/hi hello 87. Dockerfile for running a Java program $ cat Dockerfile FROM java:latest COPY HiHello.class / EXPOSE 8080 ENTRYPOINT ["java"] CMD ["HiHello"] $ docker build . Sending build context to Docker daemon 6.656 kB Step 1 : FROM java:latest ---> 264282a59a95 // ... Successfully built 60a14f519720 $ docker run -d -p 8080:8080 60a14f519720 16f6d7eca560c96b995be9f0c6d68167930ab7501451a452818e04ce29ec177f $ curl localhost:8080/hi hello 88. Pop quiz Which command do you use âto find layers and their sizesâ in an image using Docker CLI? A. Use âdocker images -layers â command B. Use âdocker layers command C. Use âdocker history command D. There is no way you can find layers and their sizes using Docker CLI - you need to use external tools 89. Pop quiz: answer To find layers and their sizes in an image using Docker CLI, useâdocker history command. $ docker history google/cadvisor IMAGE CREATED CREATED BY SIZE COMMENT 106e303be3a4 2 weeks ago /bin/sh -c #(nop) ENTRYPOINT ["/usr/bin/cadvi 0 B 2 weeks ago /bin/sh -c #(nop) EXPOSE 8080/tcp 0 B 2 weeks ago /bin/sh -c #(nop) ADD file:1bde294f31142b3dee 25.87 MB 2 weeks ago /bin/sh -c apk --no-cache add ca-certificates 17.13 MB 2 weeks ago /bin/sh -c #(nop) ENV GLIBC_VERSION=2.23-r3 0 B 2 weeks ago /bin/sh -c #(nop) MAINTAINER dengnan@google.c 0 B 3 months ago /bin/sh -c #(nop) ADD file:852e9d0cb9d906535a 4.799 MB 90. Pop quiz Which command do you use ârecreate the Dockerfile that was used to build that imageâ from a given image id/tag using Docker CLI? A. Use âdocker images -dockerfile â command B. Use âdocker build -reverse command C. Use âdocker history --no-trunc --out: command D. There is no way to recreate the Dockerfile that was used to build that image from a given image id/tag using Docker CLI 91. Pop quiz: answer There is NO way to recreate the Dockerfile that was used to build that image from a given image id/tag using Docker CLI. Think about Makefile: can you recreate the Makefile that was used to build that executable file? No. However, you can see the commands used to create the layers in the image. Pass ââno-truncâ option to âdocker historyâ command. Example: âdocker history --no-trunc google/cadvisor" Try it now! 92. Docker Volumes 93. Docker volume commands Command Description docker volume create Create a volume docker volume inspect Display detailed information on one or more volumes docker volume ls List the available volumes docker volume rm Remove one or more volumes 94. Commands for Docker volumes $ docker volume create --name myvolume myvolume $ docker volume ls local myvolume $ docker volume inspect myvolume [ { "Name": "myvolume", "Driver": "local", "Mountpoint": "/var/lib/docker/volumes/myvolume/_data", "Labels": {}, "Scope": "local" } ] $ docker volume rm myvolume myvolume 95. How to persist data? $ docker run -v /volumetesting --name="persistdata" alpine /bin/sh -c "echo testing persistence with volumes > /volumetesting/textfile.txtâ $ docker run --volumes-from=persistdata alpine /bin/sh -c "cat / volumetesting/textfile.txt" testing persistence with volumes Use -v option to âmount volumesâ 96. How to get info on volumes? $ docker volume ls DRIVER VOLUME NAME local 081bf425dd953c6b13f8e36f24540d191792e51dbd9c327eadae131ded5da432 local 3357f5522da19b47c3996db5e129b52d4be420ccec25d60d4473602cd25f473b $ docker volume inspect 081bf425dd953c6b13f8e36f24540d191792e51dbd9c327eadae131ded5da432 [ { "Name": "081bf425dd953c6b13f8e36f24540d191792e51dbd9c327eadae131ded5da432", "Driver": "local", "Mountpoint": "/var/lib/docker/volumes/ 081bf425dd953c6b13f8e36f24540d191792e51dbd9c327eadae131ded5da432/_data", "Labels": null, "Scope": "local" } ] Use âdocker volume ls and inspectâ options 97. Removing volumes $ docker volume rm 081bf425dd953c6b13f8e36f24540d191792e51dbd9c327eadae131ded5da432 081bf425dd953c6b13f8e36f24540d191792e51dbd9c327eadae131ded5da432 Use âdocker volume rmâ option 98. Removing containers with volumes ❖ When the container is removed, the volumes will not be removed. If the volumes also need to be removed, you have to use the -v option, as in: docker rm -v 99. Clean up volumes ❖ You can âclean upâ the volumes if you aren't using them. Use the command âdocker volume rm $(docker volume ls -q)â to remove all the volumes. 100. Use Flocker (data volume manager) source: https://clusterhq.com/assets/images/diagrams/diagram-1.jpg See: https://clusterhq.com/flocker/ 101. Docker Machine 102. Docker Machine $ docker-machine create --driver=virtualbox myhost Running pre-create checks... (myhost) Default Boot2Docker ISO is out-of-date, downloading the latest release... (myhost) Latest release for github.com/boot2docker/boot2docker is v1.12.2 (myhost) Downloading /Users/gsamarthyam/.docker/machine/cache/boot2docker.iso from https://github.com/ boot2docker/boot2docker/releases/download/v1.12.2/boot2docker.iso... // ... Setting Docker configuration on the remote daemon... Checking connection to Docker... Docker is up and running! To see how to connect your Docker Client to the Docker Engine running on this virtual machine, run: docker-machine env myhost $ docker-machine env myhost export DOCKER_TLS_VERIFY="1" export DOCKER_HOST="tcp://192.168.99.100:2376" export DOCKER_CERT_PATH="/Users/gsamarthyam/.docker/machine/machines/myhost" export DOCKER_MACHINE_NAME="myhost" # Run this command to configure your shell: # eval $(docker-machine env myhost $ docker-machine ls NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS myhost - virtualbox Running tcp://192.168.99.100:2376 v1.12.2 Create and manage machines running Docker (cloud or on your computer) 103. Docker Compose 104. docker-compose commands Command Description docker-compose up (Re)build services docker-compose kill Kill the containers docker-compose logs Show the logs of the containers docker-compose down Stop and remove images, containers, volumes and networks docker-compose rm Remove stopped containers 105. Creating multiple Docker containers Step 1. Create a docker-compose.yml file (or docker-compose.yaml file) Step 2. Execute âdocker-compose up -dâ Step 3. Execute âdocker-compose logsâ from another shell (but from same dir) Step 4. Execute âdocker-compose downâ 106. docker-compose commands Command Description docker-compose up (Re)build services docker-compose kill Kill the containers docker-compose logs Show the logs of the containers docker-compose down Stop and remove images, containers, volumes and networks docker-compose rm Remove stopped containers 107. Docker voting app 108. Docker voting app Step 1. Download .zip or clone: https://github.com/docker/example-voting-app Step 2. Unzip the file and go to that directory from your shell Step 3. Type âdocker-compose up -dâ Step 4. From another shell, go to the same directory & type âdocker-compose logsâ Step 5. In browser address bar, type âhttp://localhost:5000" Step 6. In browser address bar, type âhttp://localhost:5001" Step 7. From the shell, in that directory, type âdocker-compose downâ 109. Docker voting app 110. Docker Networking 111. Getting the ip address of a container $ docker inspect --format '{{ .NetworkSettings.IPAddress }}' fervent_sinoussi 172.17.0.6 $ docker attach fervent_sinoussi root@856aed6a92f1:/# ip addr // ... 92: eth0@if93: mtu 1500 qdisc noqueue state UP group default link/ether 02:42:ac:11:00:06 brd ff:ff:ff:ff:ff:ff inet 172.17.0.6/16 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::42:acff:fe11:6/64 scope link valid_lft forever preferred_lft forever root@856aed6a92f1:/# cat /etc/hosts // ... 172.17.0.6 856aed6a92f1 root@856aed6a92f1:/# There are many ways to get the IP address of a container: 1. Use the docker inspect command 2. Use ip addr command from the containerâs shell 3. Use âcat /etc/hostsâ and check the entry for the container 112. How to get port mappings of a container? $ docker run -d -p5000:5000 registry c51b984b4d64a05e924c7677f20e8c5c386e8bb53f5de0369337d31f73a7cf7e $ docker port c51b984b4d64a05e924c7677f20e8c5c386e8bb53f5de0369337d31f73a7cf7e 5000/tcp -> 0.0.0.0:5000 $ docker run -P -d nginx de6e17ededc8223c9a5ac9fee4f9493929a22a78fe88c60b643b545078c60648 $ docker port de6e17ededc8223c9a5ac9fee4f9493929a22a78fe88c60b643b545078c60648 443/tcp -> 0.0.0.0:32768 80/tcp -> 0.0.0.0:32769 $ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES de6e17ededc8 nginx "nginx -g 'daemon off" 21 seconds ago Up 20 seconds 0.0.0.0:32769->80/tcp, 0.0.0.0:32768->443/tcp reverent_wright 113. Three kinds of networks $ docker network ls NETWORK ID NAME DRIVER SCOPE a3bb9a40c8e3 bridge bridge local 399711fd0635 host host local 790ae8b43d9b none null local Default, single-host driver 114. Docker network commands Command Description docker network connect Connect a container to a network docker network create Create a network docker network disconnect Disconnect a container from a network docker network inspect Display detailed information on one or more networks docker network ls List networks docker network rm Remove one or more networks 115. Bridge network $ docker network inspect bridge [ { "Name": "bridge", "Id": "39457e56e7c0d172a239745e10ebf24f9e5046e9bc98f978f4e759f1f4e486c3", "Scope": "local", "Driver": "bridge", "EnableIPv6": false, "IPAM": { "Driver": "default", "Options": null, "Config": [ { "Subnet": "172.17.0.0/16", "Gateway": "172.17.0.1" } ] }, "Internal": false, "Containers": { "19857aa228f7ba76fc10d1e992e9fe49a0d361b5daab3a0a2703267aab862c58": { "Name": "furious_mcnulty", "EndpointID": "e2a7423a9108fa7bb18ea9893e299e1bceb73fa13c3123c5b0d515790be477d3", "MacAddress": "02:42:ac:11:00:03", "IPv4Address": "172.17.0.3/16", "IPv6Address": "" }, "eaf4697b9989666e0c79cce6dc03697c8226aea37157d97bce1d08e250fb3c36": { "Name": "cadvisor", "EndpointID": "f907c7510aaf7196e6419733b64da689330948ca0f3c88bd4dd41258a4503e42", "MacAddress": "02:42:ac:11:00:02", "IPv4Address": "172.17.0.2/16", "IPv6Address": "" } }, "Options": { "com.docker.network.bridge.default_bridge": "true", "com.docker.network.bridge.enable_icc": "true", "com.docker.network.bridge.enable_ip_masquerade": "true", "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0", "com.docker.network.bridge.name": "docker0", "com.docker.network.driver.mtu": "1500" }, "Labels": {} } ] By default, containers are added to the bridge network. You can see the containers in bridge network here 116. Pop quiz You are creating a new container with this command: docker run -d --name myubuntu ubuntu /bin/sh -c "while true; do echo current date and time is: $(date); sleep 10; doneâ Which network is the âmyubuntuâ container attached to? A. Bridge B. Overlay C. Custom D. None 117. Pop quiz: answer Bridge network. By default, a newly created container is attached to the bridge network (unless a different network is specified, for example, using the âânetworkâ option with the docker run command). $ docker network inspect bridge [ { "Name": âbridge", // ... "Containers": { "04579b88a74c981ae854261dffc7ab17328c28bb6fafec0f9c1e9431e77b3b27": { "Name": "myubuntu", "EndpointID": "8a0e7a2559eac35eb60a90e85554679de276bd1ba39ff3a4083301d08e9ee384", "MacAddress": "02:42:ac:11:00:03", "IPv4Address": "172.17.0.3/16", "IPv6Address": "" }, // ... }, // ... } ] 118. Orchestration Docker swarm provides âclustering capabilitiesâ, i.e., be able to treat group of Docker engines into a single virtual Docker engine 119. Docker Swarm âswap, plug, and playâ 120. Docker Swarm $ docker swarm init Swarm initialized: current node (81snul7czu9pg42h3qnm2v5hr) is now a manager. To add a worker to this swarm, run the following command: docker swarm join --token SWMTKN-1-2tn8fic07uivk29ith095me3r6cro7bkrfnbtv6va3qvf6urew-6olc37m9ljymly3fcb6rig2nu 192.168.65.2:2377 To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions. 121. Clustering for Docker âbatteries included but swappableâ 122. Docker Security 123. Docker workbench for security docker run -it --net host --pid host --cap-add audit_control -v /var/lib:/var/lib -v /var/run/docker.sock:/var/run/docker.sock -v /usr/lib/systemd:/usr/lib/systemd -v /etc:/etc --label docker_bench_security docker/docker-bench-security git clone https://github.com/docker/docker-bench-security.git cd docker-bench-security sh docker-bench-security.sh git clone https://github.com/docker/docker-bench-security.git cd docker-bench-security docker-compose run --rm docker-bench-security OR OR Source: https://github.com/docker/docker-bench-security 124. Docker workbench for security $ sh docker-bench-security.sh # ------------------------------------------------------------------------------ # Docker Bench for Security v1.1.0 # # Docker, Inc. (c) 2015- # # Checks for dozens of common best-practices around deploying Docker containers in production. # Inspired by the CIS Docker 1.11 Benchmark: # https://benchmarks.cisecurity.org/downloads/show-single/index.cfm?file=docker16.110 # ------------------------------------------------------------------------------ [WARN] Some tests might require root to run [INFO] 1 - Host Configuration [WARN] 1.1 - Create a separate partition for containers [PASS] 1.2 - Use an updated Linux Kernel [WARN] 1.4 - Remove all non-essential services from the host - Network [WARN] * Host listening on: 7 ports [PASS] 1.5 - Keep Docker up to date [INFO] * Using 1.12.1 which is current as of 2016-08-16 [INFO] * Check with your operating system vendor for support and security maintenance for docker [INFO] 1.6 - Only allow trusted users to control Docker daemon [WARN] 1.7 - Failed to inspect: auditctl command not found. [INFO] 1.8 - Audit Docker files and directories - /var/lib/docker [INFO] * Directory not found [INFO] 1.9 - Audit Docker files and directories - /etc/docker [INFO] * Directory not found ... [INFO] 2 - Docker Daemon Configuration [WARN] 2.1 - Restrict network traffic between containers [PASS] 2.2 - Set the logging level [PASS] 2.3 - Allow Docker to make changes to iptables [PASS] 2.4 - Do not use insecure registries [WARN] 2.5 - Do not use the aufs storage driver [INFO] 2.6 - Configure TLS authentication for Docker daemon [INFO] * Docker daemon not listening on TCP [INFO] 2.7 - Set default ulimit as appropriate [INFO] * Default ulimit doesn't appear to be set [WARN] 2.8 - Enable user namespace support [PASS] 2.9 - Confirm default cgroup usage [PASS] 2.10 - Do not change base device size until needed [WARN] 2.11 - Use authorization plugin [WARN] 2.12 - Configure centralized and remote logging [WARN] 2.13 - Disable operations on legacy registry (v1) ... 125. Docker workbench for security ❖ Use the free Docker Workbench For Security (https:// github.com/docker/docker-bench-security) to check for violations of security best practices 126. Monitoring Docker 127. Stats for all running containers Use âdocker statsâ command $ docker stats Displays resource utilisation (cpu, memory, etc) details; automatically updated when details change 128. Stats for a specific Docker Use âdocker stats â command $ docker stats sleepy_wescoff CONTAINER CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS sleepy_wescoff 0.00% 1.031 MiB / 4 MiB 25.78% 648 B / 648 B 0 B / 0 B 1 129. Printing containers names in stat $ docker stats $(docker ps --format={{.Names}}) 130. Monitoring Docker Use the free cAdvisor tool (https://github.com/google/cadvisor) Now open localhost:8080 in your browser docker run --volume=/:/rootfs:ro --volume=/var/run:/var/run:rw âvolume=/ sys:/sys:ro --volume=/var/lib/docker/:/var/lib/docker:ro --publish=8080:8080 --detach=true --name=cadvisor google/cadvisor:latest 131. Monitoring Docker localhost:8080 132. Monitoring Docker ❖ datadog (https://www.datadoghq.com/) ❖ sysdig (http://www.sysdig.org/) ❖ prometheus (https://prometheus.io/) 133. Other topics 134. How do debug on a running container? $ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 9128bf57e03c ubuntu "/bin/sh -c 'while tr" 24 minutes ago Up 24 minutes lonely_einstein $ docker exec -ti lonely_einstein /bin/bash root@9128bf57e03c:/# Use âdocker execâ command 135. Can I use GUI instead of command-line? Use âkitematicâ (https://github.com/docker/kitematic) 136. Different ways to access Docker Command-Line Interface (CLI) Graphical User Interface (Kitematic) Representational State Transfer (REST) API 137. Crazy stuff: Docker in Docker!! Use âdocker run --privileged -d docker:dind" âdocker:dindâ is the official âDocker in Docker base imageâ See: https://github.com/jpetazzo/dind 138. Myths and Misconceptions 139. Docker *completely* replaces VMs 140. Containers AND VMs 141. Docker is *completely* portable There are limitations to portability with Docker (depending on what you mean by âportableâ). For example, you can run a Windows Docker container only on Windows and run a Linux Docker container only on Linux (and not vice versa). Build once, run anywhere - but conditions apply! 142. âManagement says we need Docker, so letâs use itâ 143. Quick reference 144. Docker commands attach Attach to a running container build Build an image from a Dockerfile commit Create a new image from a container's changes cp Copy files/folders between a container and the local filesystem create Create a new container deploy Create and update a stack from a Distributed Application Bundle (DAB) diff Inspect changes on a container's filesystem events Get real time events from the server exec Run a command in a running container export Export a container's filesystem as a tar archive history Show the history of an image images List images import Import the contents from a tarball to create a filesystem image info Display system-wide information inspect Return low-level information on a container, image or task kill Kill one or more running container load Load an image from a tar archive or STDIN login Log in to a Docker registry. logout Log out from a Docker registry. logs Fetch the logs of a container network Manage Docker networks node Manage Docker Swarm nodes pause Pause all processes within one or more containers plugin Manage Docker plugins 145. Docker commands port List port mappings or a specific mapping for the container ps List containers pull Pull an image or a repository from a registry push Push an image or a repository to a registry rename Rename a container restart Restart a container rm Remove one or more containers rmi Remove one or more images run Run a command in a new container save Save one or more images to a tar archive (streamed to STDOUT by default) search Search the Docker Hub for images service Manage Docker services stack Manage Docker stacks start Start one or more stopped containers stats Display a live stream of container(s) resource usage statistics stop Stop one or more running containers swarm Manage Docker Swarm tag Tag an image into a repository top Display the running processes of a container unpause Unpause all processes within one or more containers update Update configuration of one or more containers version Show the Docker version information volume Manage Docker volumes wait Block until a container stops, then print its exit code 146. Source: http://zeroturnaround.com/wp-content/uploads/2016/03/Docker-cheat-sheet-by-RebelLabs.png 147. Where to learn more? 148. Relevant URLs ❖ Detailed list of resources: https://github.com/hangyan/ docker-resources ❖ Self-learning courses: https://training.docker.com/ ❖ Detailed documentation: https://docs.docker.com/ ❖ Detailed tutorial (presentation): http://docker.training ❖ SE-Radio Episode 217: James Turnbull on Docker ❖ Docker related presentations in parleys.com ❖ Blog on Docker resource utilisation 149. DOCKER: UP & RUNNING ➤ Covers how to develop, test, debug, ship, scale, and support with Docker from DevOps perspective ➤ We liked the useful tips; examples: ➤ âMaximize robustness with fast startup and graceful shutdown.â ➤ âExplicitly declare and isolate dependencies.â ➤ âStrictly separate build and run stages.â http://amzn.com/1491917571 âDocker: Up & Runningâ, Karl Matthias, Sean P. Kane, O'Reilly Media; 1 edition (July 3, 2015) 150. THE DOCKER BOOK ➤ Interesting sub-title: âContainerization is the new virtualizationâ. ➤ From James Turnbull (CTO at Kickstarter and Advisor at Docker) ➤ Useful to get comfortable with core concepts of Docker ➤ Useful for developers, operations staff (and DevOps), and SysAdmins ➤ Supporting website: http:// dockerbook.com/http://www.amazon.in/dp/B00LRROTI4 The Docker Book, James Turnbull, Amazon Digital South Asia Services, July 2014 151. DOCKER COOKBOOK ➤ Contents written in recipe format (Problem, Solution, Discussion) ➤ Useful because we can look for solutions to the problems that we face when using Docker ➤ What we like: it covers topics that are not covered well in other books including Kubernetes, Docker ecosystem tools, monitoring Docker, and application use cases (CI, CD) http://amzn.com/149191971X âDocker Cookbookâ, SÃbastien Goasguen, O'Reilly Media, 2015 152. DOCKER IN ACTION ➤ Wide coverage from basics to advanced topics like managing massive clusters ➤ Book organised into three parts: ➤ Keeping a tidy computer ➤ Packaging software for distribution ➤ Multi-container and multi-host environments ➤ The third part is more interesting for us because it is not covered well in other books ➤ Covers Docker Compose, Machine and Swarm http://amzn.com/1633430235 Docker in Action, Jeff Nickoloff, Manning Publications, 2016 153. USING DOCKER ➤ Book organised into three parts: ➤ Background and Basics ➤ The Software Lifecycle with Docker ➤ Tools and Techniques ➤ Useful example: Walks you through the steps to develop and deploy web applications with Docker ➤ Though the book touches upon basics, it covers more advanced topics http://amzn.com/1491915765 Using Docker: Developing and Deploying Software with Containers, Adrian Mouat, O'Reilly Media, 2016 154. Upcoming bootcamps AngularJS (22nd Oct) Modern Software Architecture (5th Nov) SOLID Principles (19th Nov) 155. Meetups h"p://www.meetup.com/JavaScript-Meetup-Bangalore/ h"p://www.meetup.com/Container-Developers-Meetup-Bangalore/ h"p://www.meetup.com/So>ware-Cra>smanship-Bangalore-Meetup/ h"p://www.meetup.com/Core-Java-Meetup-Bangalore/ h"p://www.meetup.com/Technical-Writers-Meetup-Bangalore/ h"p://www.meetup.com/CloudOps-Meetup-Bangalore/ h"p://www.meetup.com/Bangalore-SDN-IoT-NetworkVirtualizaHon-Enthusiasts/ h"p://www.meetup.com/So>wareArchitectsBangalore/ 156. ganesh@codeops.tech @GSamarthyam www.codeops.tech slideshare.net/sgganesh +91 98801 64463 bit.ly/ganeshsg 157. Image credits ❖ https://pbs.twimg.com/media/CH-ISJGUwAAt8hQ.png ❖ http://patg.net/assets/container_vs_vm.jpg ❖ http://static1.businessinsider.com/image/525e9c7669bedd9c3015dc60-1190-625/the-10-funniest-dilbert-comic-strips-about-idiot-bosses.jpg ❖ https://blog.docker.com/wp-content/uploads/2014/03/docker-execdriver-diagram.png ❖ https://docs.docker.com/engine/article-img/architecture.svg ❖ https://en.wikipedia.org/wiki/File:Docker-linux-interfaces.svg ❖ http://lohmander.me/content/images/2015/10/d2f.jpg ❖ https://camo.githubusercontent.com/ ec87adde4b3711198fb90ff112eb4361d313e067/68747470733a2f2f73332e616d617a6f6e6177732e636f6d2f7765622d6172746566616374732f6361727 46f6f6e2d7768616c652d382e6769662b28343030254333253937323235292e706e67 ❖ http://blog.gutcheckit.com/hubfs/Headers/Blogs/Q215-Blog-KernelSeasonsRecap-Header-060415.jpg ❖ http://core0.staticworld.net/images/article/2014/11/docker_linux-100530817-primary.idge.jpg ❖ http://cdn.hrpayrollsystems.net/wp-content/uploads/2015/02/best-practices-hris.jpg ❖ https://blog.docker.com/media/2015/07/moby_art.png ❖ https://blog.docker.com/media/2015/04/sticker-02-15-2-1024x711.png ❖ http://blogs-images.forbes.com/janakirammsv/files/2016/06/docker1.jpg?width=960 ❖ http://blogs-images.forbes.com/janakirammsv/files/2016/06/Docker_CI_CD.jpg?width=960 158. Image credits ❖ http://cormachogan.com/wp-content/uploads/2016/07/docker-volumes.jpg ❖ http://image.slidesharecdn.com/swarmonlinemeetup-150507153718-lva1-app6891/95/docker- swarm-020-5-638.jpg?cb=1431013147 ❖ http://image.slidesharecdn.com/swarmonlinemeetup-151111212937-lva1-app6892/95/docker-online- meetup-28-productionready-docker-swarm-11-638.jpg?cb=1447459032 ❖ https://blog.docker.com/media/2015/04/docker-turtles-communication.jpg ❖ https://pbs.twimg.com/media/CtSCE2FUEAA94Pd.jpg ❖ https://i0.wp.com/blog.docker.com/wp-content/uploads/3-1.png?w=560&ssl=1 ❖ https://blog.docker.com/media/2015/11/logo-title-final-swarm-2d.png ❖ http://image.slidesharecdn.com/docker-swarm-mike-goelzer-mv-meetup-45min- workshop022420161-160228024416/95/docker-swarm-docker-native-clustering-5-638.jpg?cb=1456856097 ❖ http://54.71.194.30:4110/engine/reference/api/images/event_state.png/ ❖ http://edge.alluremedia.com.au/m/l/2015/05/DockerExploration.jpg ❖ https://www.docker.com/sites/default/files/home-1-solutions-2_0.jpg 159. Image credits ❖ https://i2.wp.com/blog.docker.com/wp-content/uploads/windows.png?resize=975%2C546&ssl=1 ❖ http://taylorholmes.com/wp-content/uploads/2010/08/totem11-1024x364.jpg ❖ https://cdn-images-2.medium.com/max/2000/1*k8n7Jx9UaLRAxum9HMp8nQ.png ❖ https://pbs.twimg.com/media/CpA4RzoXEAAf83a.png ❖ http://thenewstack.io/wp-content/uploads/2016/02/Docker.png ❖ https://lh3.googleusercontent.com/-4Cpex5VrtFM/Vl4mKLq5FbI/AAAAAAAAAxE/FJRVex2O6tE/w485-h370/ docker_monstro.png ❖ https://www.cloudbees.com/sites/default/files/jenkins-docker-cd-express.jpg ❖ http://jbu.io/wp-content/uploads/2015/10/docker.jpg ❖ https://encrypted-tbn2.gstatic.com/images?q=tbn:ANd9GcRiV1MXhBG39oQPuyVyAF5ZMaYzi3pOYvm6pHeJA71x8PrfVD7p ❖ http://2.bp.blogspot.com/-0qweEK2XCg8/VhQS0dffOTI/AAAAAAAAArw/gpOGuJELCP4/s1600/docker.png ❖ http://momentumtelecom.com/wp-content/uploads/2014/10/training-icons-qrg.png ❖ https://learning-continuous-deployment.github.io/assets/images/compose.jpg ❖ http://www.showroomworkstation.org.uk/pictures/Logos/~jDKjyjDDDDKjyjU6/Try_This_sm.png https://www.friendbookmark.com/videos/979/docker-networking-basics-coupling-with-software-defined-networksThis presentation reminds Docker networking, exposes Software Defined Network basic paradigms, and then proposes a mixed-up implementation taking benefits of a coupled use of these two technologies. Implementation model proposed could be a good starting point to create multi-tenant PaaS platforms. As a bonus, OpenStack Neutron internal design is presented. You can also have a look on our previous presentation related to enterprise patterns for Docker: http://fr.slideshare.net/ArnaudMAZIN/docker-meetup-paris-enterprise-docker Topics covered: 1. 1 TÃl : +33 (0)1 58 56 10 00 Fax : +33 (0)1 58 56 10 01 www.octo.com OCTO 2013 50, avenue des Champs-ElysÃes 75008 Paris - FRANCE Follow us on Twitter! @AdrienBlind @ArnaudMazin 2. 2 REMINDER: DOCKER NETWORKING Agenda INTRODUCTION TO SOFTWARE DEFINED NETWORKS MIX UP 1 TAKE AWAY 2 4 3 3. 3 Reminder: Docker networking 4. 4 Host Natively, each container runs isolated from the outside world A bridge network (an internal, virtual switch) is provided with Docker but does not allow containerized processes to communicate together without further config (see next slides) Isolation by design Container âappâ Bridge docker0 Container âdbâ Tomcat (port 8080) MySQL (port 3306) 5. 5 Exposing ports allows containers services to talk together through the bridge network Dockerfile: add line âEXPOSE â (several ports can be appended) Command line switch: âdocker run ... âexpose â Example for the âappâ container: âdocker run ... âexpose 8080â Exposing ports Host Container âappâ Bridge docker0 Container âdbâ Tomcat (port 8080) MySQL (port 3306) TCP 3306 exposed TCP 8080 exposed  The app is able to talk to its database  6. 6 Linking enables a client container to get information related to a resource container (also known as the linked container) Command line switch: âdocker run -link â Example on âappâ container: âdocker ârun â âlink db:dbaliasâ Linking containers Host Container âappâ Bridge docker0 Container âdbâ Tomcat (port 8080) MySQL (port 3306) Env variables DBALIAS_PORT_3306_TCP=tcp://1.2.3.4:3306 DBALIAS_PORT_3306_TCP_PROTO=tcp DBALIAS_PORT_3306_TCP_ADDR=1.2.3.4 DBALIAS_PORT_3306_TCP_PORT=3306  The app knows where to join its database  7. 7 Mapping ports enable to publish containerâs ports on âexternal interfacesâ. Ports may be translated. Command line switch: âdocker run -p â Example for âappâ container: âdocker ârun â âp 80:8080â Mapping ports Host Container âappâ Bridge docker0 Container âdbâ Tomcat (port 8080) MySQL (port 3306)  The app is reachable from the external network at port 80 of the host, while the database container only remains reachable to other containers  IP_interface TCP 80 map Tomcatâs 8080 port External network 8. 8 These characteristics enables containers: To either remain fully isolated or talk to all other containers of the same host having their services exposed To map one-by-one services on the hosts interface Hmmâ What if I want to isolate groups of containers? Conclusion 9. 9 Introduction to Software Defined Networks 10. 10 Traditional datacenter management VM VM VM VM VM VM VM VM VM Internet Internet DMZ Physical overview Logical overview Tenant #1 Tenant #3 Tenant #2 LAN DMZ LAN DMZ1 DMZ2 Logical topologies are quite different from physical ones These designs traditionally relies a lot on low network layers (L2, etc.) 11. 11 Multi datacenter perspective VM VM VM VM VM VM VM VM VM Physical overview VM VM VM VM VM VM VM VM VM â Limited to 2048 VLANs â Lack of dynamic provisioning â Involves subnetting or encapsulation to flow over L3 networks Internet 12. 12 SDNs proposes network solutions embracing cloud paradigms Massively multi-tenant Thousands tenants, massively scalable Easy & fast (de)provisioning Infra as code, API centric Infrastructure agnostic L3, does not stick with lower levels (physical designs, vlans & co) Decouple infrastructure & tenants lifecycles Cross technology, vendor agnostic SDNs value proposal 13. 13 Enables to abstract networking topologies & services from wiring and equipments Centralize network intelligence Standardized management APIs: ex. OpenFlow for both physical & virtual network equipments SDNs overview (Source Wikipedia) 14. 14 On hosts (hypervisors, Docker hostsâ), SDNs mostly rely on: Allocating network bridges (virtual and internal switches) Setting ACLs to decide which flow to allow or deny Connecting these bridges to external world through real hostâs interfaces Host perspective 15. 15 Network perspective â Low level isolation Host #3 SDN network #1 SDN network #1 Host #1 SDN network #1 SDN network #1 Host #2 SDN network #1 L2-Network focused Keep traditional paradigms but use API/centralize intelligence Requires VLANs, VPLSâ to spread same virtual networks accross several hosts Enforces low-level isolation Consider API-based network configuration (ie. OpenFlow) to centralize and facilitates network management, making it more dynamic VPLS Router Physical net. VPLS network The link tags both virtual network trafics in vlans The VPLS ensure providing L2 networks at all ends â strong req.! 16. 16 Full-mesh, network agnostic and encapsulated approach Relies on L3 networks and GRE, VXLANâ tunnels for inter-hosts communication (avoid using L2) Network agnostic, til hosts can route trafic SDN Routers must route traffic between an inner virtual net and the ext. world Network perspective â Full mesh Host #3 Host #1 Host #2 SDN network #1 SDN network #1 SDN network #1 SDN network #1 SDN network #1 Router Physical net. Flow encapsulation L3 network 17. 17 Mix up 18. 18 How to support several containers related to a same tenant, accross multiple hosts (or even multiple datacenters or providers), and avoid them to talk to other containers in the same time ? Answer to this question enables several usecases Isolate containers of a same app without having to face limits of a single Docker host Resilience (ex. spread an app server farm over multiple hosts) Multi providers (ex. spread containers over several clouds/hosters, overflow mgmtâ) â Usecases â Back to Docker Internet LAN DMZ DMZ DMZ 1 DMZ 2 LAN Container Tenant #1 Tenant #3 Tenant #2 19. 19 Host #1 Host #3Host #2 0. From where we start Container âappâ Container âdbâ Bridgedocker0 Container âappâ Container âdbâ Bridgedocker0 Container âappâ Container âdbâ Container âappâ Bridgedocker0 Container âappâ TCP ports of services running in containers are mapped Get rid of actual Docker bridges implementation ! If not, all containers will talk together across a same host 20. 20 Use OpenVSwitch to create bridges on each host, for each tenantâs subnet. For instance, on host #1:  ovs-vsctl add-br tech-br  ovs-vsctl add-port tech-br tep0 -- set interface tep0 type=internal  ifconfig tep0 192.168.1.1 netmask 255.255.255.0  ovs-vsctl add-br sdn-br0  ovs-vsctl set bridge sdn-br0 stp_enable=true 1. Create SDN compliant bridges Host #1 sdn-br0 Host #3 sdn-br0 sdn-br1 Host #2 sdn-br0 sdn-br1 Simplified view. Detailed insight exposed in later slides Once per host: common plumbing (detailed slide 24) For each bridge: create and protect against ethernet loops using Spanning Tree Protocol (beware, in complex/large deployments, it may consumes a lot of CPU!) 21. 21 2. Link SDN bridges Host #1 sdn-br0 Host #3 sdn-br0 sdn-br1 Host #2 sdn-br0 sdn-br1 Use OpenVSwitch to link corresponding bridges accross hosts In this example, we decided to use the full-mesh approach. On host #1: ovs-vsctl add-port sdn-br0 gre0 --set interface gre0 type=gre options:remote_ip:1.2.3.2 ovs-vsctl add-port sdn-br0 gre1 --set interface gre1 type=gre options:remote_ip:1.2.3.3 1.2.3.1 1.2.3.2 1.2.3.3 Simplified view. Detailed insight exposed in later slides 22. 22 U 3. Instanciate VMs and map them to the bridges Host #1 sdn-br0 Host #3 sdn-br0 sdn-br1 Host #2 sdn-br0 sdn-br1 Container âappâ Container âdbâ Container âappâ Container âdbâ Container âappâ Container âappâ Container âdbâ Container âappâ Container âdbâ Instanciate containers without pre-built interfaces to avoid plugging containers to native docker0 bridge Use âdocker run â -n=falseâ switch in âdocker runâ calls 23. 23 U 4. Connect containers to the appropriate bridge Host #1 sdn-br0 Host #3 sdn-br0 sdn-br1 Host #2 sdn-br0 sdn-br1 Container âappâ Container âdbâ Container âappâ Container âdbâ Container âappâ Container âappâ Container âdbâ Container âappâ Container âdbâ Use pipework.sh from J. Petazzoni to easily plug containers to a chosen bridge https://github.com/jpetazzo/pipework Creates network adapter in each containers, allocate them an IP (manually/static vs DHCP), and plug it to the bridge. Per container: âpipework bridge_name $container_id container_ip/24â 24. 24 Switch Insight - Detailed view between two hosts Previous drawings were simplified to ease overall understanding Following schema details more deeply what happened inside a single host Host #1 sdn-br0 Container Host #2 Container gre 0 tech-br tep0 eth0 sdn-br0 Container Container gre 0 tech-br tep0 eth0 Switch ovs-vsctl add-br tech-br ovs-vsctl add-port tech-br tep0 -- set interface tep0 type=internal ovs-vsctl add-port sdn-br0 gre0 --set interface gre0 type=gre options:remote_ip:1.1.1.1 GRE tunnel in which the traffic really flows Ethernet link between an adapter and a switch L2 switch Network adapter pipework sdn-br0 $container_id 192.168.0.3/24 eth1 eth1ovs-vsctl add-br sdn-br0eth1 eth1 1.1.1.1/24 2.2.2.2/24 192.168.1.2/24 192.168.0.3/24192.168.0.2/24192.168.0.1/24 192.168.0.4/24 4 7 6 1 2 ifconfig tep0 192.168.1.1 netmask 255.255.255.0 3 L3 routed network Docker container Docker host pipework sdn-br0 $container_id 192.168.0.4/24 8 192.168.1.1/24 Repeat step #6 to create additional tunnels in order to reach other hosts ovs-vsctl set bridge sdn-br0 stp_enable=true 5Virtual, direct link established through the GRE tunnel 25. 25 Bonus: Alternate design - OpenStack Neutron paradigms Alternate design, based on OpenStack Neutron paradigms â notice that use of VLAN limits to 2048 tenant networks on each hosts All VMs/containers of a same tenant network are segregated inside a dedicated, local VLAN of a shared unique bridge Full-mesh of GRE tunnels between all hosts On each host, local mapping between a local tenant network VLAN and its GRE identifier shared across all hosts Full story from VM A to B (tenant #1): traffic outgoing A is tagged with VLAN 20 entering br-int, flows to br-tun, is untagged entering GRE tunnel while the GRE identifier 11111 is set on transmitted packets. At the other end of the GRE tunnel, the traffic having the GRE id 11111 is assigned to VLAN 40, then flow outside br-tun, to br-int, and is finally untagged before entering B. Full story from A to C (tenant #1): traffic is tagged with VLAN 20 entering br-int, then flows to br-eth1 which finally untag inbound trafic (or assign a different VLAN corresponding to the external world) Switch Host #1 br-int br-eth0 eth0 Switch GRE tunnel in which the traffic really flows Ethernet link between an adapter and a switch L2 switch Network adapter L3 routed network VM or container Host/server br-eth1 eth1 br-tun Tenant #2 Local VLAN 30 Tenant #1 Local VLAN 20 A single bridge is used for all VMs/containers ; VMs of different tenants isolated using local VLANs (not exposed externally !) A single bridge is used as end-points for all GRE tunnels used for full-mesh One bridge is created for each physical interface Host #2 br-int br-eth0 eth0 br-tun Tenant #2 Local VLAN 50 Tenant #1 Local VLAN 40 Switch or VLAN (not related to internal VLANs) gre 0 gre 0 A B Server C Flow rules: Tenant #1: VLAN 20  GRE ID 11111 Tenant #2: VLAN 30  GRE ID 11112 Single patch link between the two bridges supporting all traffic from/to full-mesh Flow rules: Tenant #1: VLAN 40  GRE ID 11111 Tenant #2: VLAN 50  GRE ID 11112 26. 26 Take away 27. 27 Use Docker for containers hosting, externalize SDN management Disable bridges management features in Docker, use OpenVSwitch Abstract from low level network considerations whenever possible, between hosts (L2 VLANs for instances): consider tunneling Get further Use OpenStack Neutron to centralize & automatize the whole network conf. You definitively use VLANs ? Consider encapsulating several VLANs in your own tenant network Other (dirty) options: Docker in VMs nesting Multiple Docker instances on the same host ebtables/iptables/openvswitch acls on flat network Take away 28. 28 https://www.friendbookmark.com/videos/978/docker-on-openstack-by-opensource-consultingIt's presentation for technet 2015 in korea. I changed the format to pptx, 목차는 아래와 같습니다. Openstack 인프라 구축 (4 node 구성) [ 30분] Openstack 위에 VM 생성 [ 20분 ] docker 구축 기초 [ 30분] 오픈스택에 docker를 연결 [ 30분] Docker로 WEB서비스 구축 [ 15분] Openstack 위에 Docker로 WEB서비스 구축 [ 15분] Docker로 jenkins 구현 [30분] Topics covered: 1. 1 2015. 3. 12 ㈜ 오픈소스컨설팅 김호진 Docker on Openstack http://www.slideshare.net/osc_hojinkim/docker-on-openstack-by-osc 2. 2http://www.redbooks.ibm.com/redbooks/pdfs/sg248199.pdf mainframe UNIX LINUX 25- 2개 3. 3 Contents 3. docker 구축 기초 [ 30분] 4. 오픈스택에 docker를 연결 [ 30분] 1. Openstack 인프라 구축 (4 node 구성) [ 30분] 2. Openstack 위에 VM 생성 [ 20분 ] 5. Docker로 WEB서비스 구축 [ 30분] 6. 7. Docker로 jenkins 구현 [15분] Openstack 위에 Docker로 WEB서비스 구축 [ 15분] 4. 4 왜 Docker on Openstack 인가? OpenStack / KVM / Docker Openstack은 전반적인 datacenter 운영 KVM 기반 가상화는 컴퓨터 자원관리 측면 Docker는 어플리케이션 배포관련 컨테이너 http://docs.openstack.org/juno http://behindtheracks.com/category/juno/ Openstack은 클라우드 인프라 스트럭처에서 제공해 주는 멀티테넌트의 보안 및 격리, 관리 및 모니터링, 스토리지 및 네트워킹등은 전반적인 datacenter운영 기반 오픈스택위에 리소스 개수에 따른 매모리 사용률 성능비교표 Docker는 높은 이식성, 하드웨어, Framework 독립적인 컨테이너. 속도 / 효율성/ 이동성 더 적은 메모리/CPU OVERHEAD Kvm/vmwaere/virtual machine 어디든 도커라는것을 인식하지 못한채 리눅스 컨테이너 관리가능 5. 5 What to do during 30 min. OpenStack Juno Install with Neutron on CentOS 7 3 nodes configuration (default) 3 networks configuration ( tunnel=>gre) Local cinder ( limits of Test Bed), but disk was divided to another disk (uses alone) http://docs.openstack.org/juno http://behindtheracks.com/category/juno/ 192.168.0.0/24 6. 6 What to do during 30 min. OpenStack Juno Install with Neutron on CentOS 7 후에 최종적으로 4 node configuration으로 변환될 예정임. (+1 compute node) http://docs.openstack.org/juno http://behindtheracks.com/category/juno/ 첫번째 eth0에는 각기 아래 ip가 부여됩니다.(Mgt network) â juno-controller: 192.168.32.181 / juno-network: 192.168.32.182 â juno-compute01: 192.168.32.183 / juno-compute02: 192.168.32.184 두번째 eth1에는 각기 아래 ip가 부여됩니다. (tunneling network) * juno-network: 192.168.33.182 / juno-compute01: 192.168.33.183 / juno-compute02: 192.168.33.184 세번째 eth2에는 floating IP를 가지게 됩니다. (public network-floating) * juno-network: pulbic IP는 따로 IP를 주지 않음. 7. 7 openstack 구축시 log 보는법 conf 파일에 verbose=true 옵션을 걸어 더 상세히 볼수 있음. openstack의 대부분 내용은 /var/log/messages에 나옴. verboase=ture option을 주면 /var/log/messages가 각자 원하는 /var/#service/#service.log가 나옴 . 그리고 /var/log/messages에 error뿐 아니라 동작 내용까지 모두 기록됨. http://docs.openstack.org/juno http://behindtheracks.com/category/juno/ 기존 log ( /var/log/cinder/scheduler.log) 2015-03-13 03:09:12.360 1148 INFO oslo.messaging._drivers.impl_rabbit [req-844f54aa-6201-4fc4-b321-c6ab2012c296 - - - - -] Connecting to AMQP server on 192.168.32.181:5672 2015-03-13 03:09:12.433 1148 ERROR oslo.messaging._drivers.impl_rabbit [req-844f54aa-6201-4fc4-b321-c6ab2012c296 - - - - -] AMQP server on 192.168.32.181:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 3 seconds. verbose=true option 적용시 (/var/log/cinder/scheduler.log) 2015-03-13 06:20:18.812 18581 INFO cinder.service [-] Starting cinder-scheduler node (version 2014.2.1) 2015-03-13 06:20:18.816 18581 INFO oslo.messaging._drivers.impl_rabbit [req-1d1a9b9c-3658-4f76-8dc1-3d74b2028a36 - - - - -] Connecting to AMQP server on 192.168.32.181:5672 2015-03-13 06:20:18.837 18581 INFO oslo.messaging._drivers.impl_rabbit [req-1d1a9b9c-3658-4f76-8dc1-3d74b2028a36 - - - - -] Connected to AMQP server on 192.168.32.181:5672 2015-03-13 06:20:19.291 18581 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 192.168.32.181:5672 2015-03-13 06:20:19.303 18581 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 192.168.32.181:5672 2015-03-13 06:20:50.814 18581 WARNING cinder.scheduler.host_manager [req-00223525-0c03-4c5d-ae9b-690ae0a10e72 d13d86ad609d4a9a8d9a84b36b954a69 3c40224ㅁ5243f443ebc2aa39605641be1 - - -] volume service is down. (host: juno- compute) 2015-03-13 06:20:50.814 18581 WARNING cinder.scheduler.filter_scheduler [req-00223525-0c03-4c5d-ae9b-690ae0a10e72 d13d86ad609d4a9a8d9a84b36b954a69 3c402245243f443ebc2aa39605641be1 - - -] No weighed hosts found for volume with properties: {} 2015-03-13 06:20:50.816 18581 ERROR cinder.scheduler.flows.create_volume [req-00223525-0c03-4c5d-ae9b-690ae0a10e72 d13d86ad609d4a9a8d9a84b36b954a69 3c402245243f443ebc2aa39605641be1 - - -] Failed to run task cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create: No valid host was found. No weighed hosts available 8. 8 openstack 구축시 log 보는법 conf 파일에 verbose=true 옵션을 걸어 더 상세히 볼수 있음. http://docs.openstack.org/juno http://behindtheracks.com/category/juno/ verbose=true option 적용시 (/var/log/messages) Mar 13 06:20:50 juno-controller cinder-api: 2015-03-13 06:20:50.230 18615 INFO cinder.api.v1.volumes [req- 00223525-0c03-4c5d-ae9b-690ae0a10e72 d13d86ad609d4a9a8d9a84b36b954a69 3c402245243f443ebc2aa39605641be1 - - -] Create volume of 2 GB Mar 13 06:20:50 juno-controller cinder-api: 2015-03-13 06:20:50.620 18615 INFO oslo.messaging._drivers.impl_rabbit [req-00223525-0c03-4c5d-ae9b-690ae0a10e72 d13d86ad609d4a9a8d9a84b36b954a69 3c402245243f443ebc2aa39605641be1 - - -] Connecting to AMQP server on 192.168.32.181:5672 Mar 13 06:20:50 juno-controller cinder-api: 2015-03-13 06:20:50.643 18615 INFO oslo.messaging._drivers.impl_rabbit [req-00223525-0c03-4c5d-ae9b-690ae0a10e72 d13d86ad609d4a9a8d9a84b36b954a69 3c402245243f443ebc2aa39605641be1 - - -] Connected to AMQP server on 192.168.32.181:5672 Mar 13 06:20:50 juno-controller cinder-api: 2015-03-13 06:20:50.686 18615 INFO cinder.api.v1.volumes [req- 00223525-0c03-4c5d-ae9b-690ae0a10e72 d13d86ad609d4a9a8d9a84b36b954a69 3c402245243f443ebc2aa39605641be1 - - -] vol={'migration_status': None, 'availability_zone': 'nova', 'terminated_at': None, 'reservations': ['01680237-32b1-4bcb-a3d4-c17b46837298', 'dd9280a1-7232-4aba-acf0-23aef02c34a9'], 'updated_at': None, 'provider_geometry': None, 'replication_extended_status': None, 'replication_status': 'disabled', 'snapshot_id': None, 'ec2_id': None, 'mountpoint': None, 'deleted_at': None, 'id': '37d5a43a-3f6c-4880-91c6- 7fba7c434211', 'size': 2, 'user_id': u'd13d86ad609d4a9a8d9a84b36b954a69', 'attach_time': None, 'source_replicaid': None, 'attached_host': None, 'display_description': None, 'volume_admin_metadata': [], 'project_id': u'3c402245243f443ebc2aa39605641be1', 'launched_at': None, 'scheduled_at': None, 'status': 'creating', 'volume_type_id': None, 'deleted': False, 'provider_location': None, 'host': None, 'consistencygroup_id': None, 'source_volid': None, 'provider_auth': None, 'display_name': u'test2', 'instance_uuid': None, 'bootable': False, 'created_at': datetime.datetime(2015, 3, 13, 10, 20, 50, 562087), 'attach_status': 'detached', 'volume_type': None, 'consistencygroup': None, 'volume_metadata': [], '_name_id': None, 'encryption_key_id': None, 'replication_driver_data': None, 'metadata': {}} Mar 13 06:20:50 juno-controller cinder-api: 2015-03-13 06:20:50.700 18615 INFO cinder.api.openstack.wsgi [req-00223525-0c03-4c5d-ae9b-690ae0a10e72 d13d86ad609d4a9a8d9a84b36b954a69 3c402245243f443ebc2aa39605641be1 - - -] http://192.168.32.181:8776/v1/3c402245243f443ebc2aa39605641be1/volumes returned with HTTP 200 9. 9 Env setting 사전 환경 조성 ( all node에서 실행) [root@juno-controller ~]# cat /etc/default/grub GRUB_TIMEOUT=5 GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)" GRUB_DEFAULT=saved GRUB_DISABLE_SUBMENU=true GRUB_TERMINAL_OUTPUT="console" GRUB_CMDLINE_LINUX="rd.lvm.lv=centos/swap vconsole.font=latarcyrheb-sun16 rd.lvm.lv=centos/root crashkernel=auto vconsole.keymap=us net.ifnames=0 rhgb quiet" GRUB_DISABLE_RECOVERY="true" Ethernet name을 익숙한 eth#으로 변경한다. cat /etc/default/grub [root@juno-controller ~]# sudo gurb2-mkconfig -o /boot/grub2/grub.cfg systemctl stop NetworkManager systemctl disable NetworkManager NetworkManger stop (추가 설치 : yum install ây net-tools) 서버 : controller 서버에 자기 정보 기입 restrict 192.168.32.0 mask 255.255.255.0 nomodify notrap server 127.127.1.0 iburst # local clock NTP설정 (yum install ntp)  /etc/ntp.conf 나머지 2 node ( network/compute node) server 192.168.32.181 iburst # local clock systemctl start ntpd.service systemctl enable ntpd.service 10. 10 Env setting 환경 변수 setting (env parameter setting) Centos 7 minimum installation yum -y install epel-release yum -y install http://rdo.fedorapeople.org/openstack-juno/rdo-release-juno.rpm yum -y upgrade #Updated: centos-release.x86_64 0:7-0.1406.el7.centos.2.6 systemctl stop firewalld.service;systemctl disable firewalld.service getenforce sed -i 's/enforcing/permissive/g' /etc/selinux/config echo 0 > /sys/fs/selinux/enforce CONTROLLER_IP=192.168.32.181 ADMIN_TOKEN=ADMIN SERVICE_PWD=service ADMIN_PWD=password META_PWD=meta123 #juno-controller node THISHOST_NAME=juno-controller THISHOST_IP=192.168.32.181 THISHOST_NETMASK=255.255.255.0 THISHOST_GATEWAY=192.168.32.1 THISHOST_DNS=192.168.32.1 THISHOST_TUNNEL_IP=na THISHOST_TUNNEL_NETMASK=24 OpenStack용 rpm을 제공하는 Third Party Repository를 모든 노드에 설정 CONTROLLER_IP=192.168.32.181 ADMIN_TOKEN=ADMIN SERVICE_PWD=service ADMIN_PWD=password META_PWD=meta123 #juno-network node THISHOST_NAME=juno-network THISHOST_IP=192.168.32.182 THISHOST_NETMASK=255.255.255.0 THISHOST_GATEWAY=192.168.32.1 THISHOST_DNS=192.168.32.1 THISHOST_TUNNEL_IP=192.168.33. 182 THISHOST_TUNNEL_NETMASK=24 CONTROLLER_IP=192.168.32.181 ADMIN_TOKEN=ADMIN SERVICE_PWD=service ADMIN_PWD=password META_PWD=meta123 #juno-compute node THISHOST_NAME=juno-compute THISHOST_IP=192.168.32.183 HISHOST_NETMASK=255.255.255.0 THISHOST_GATEWAY=192.168.32.1 THISHOST_DNS=192.168.32.1 THISHOST_TUNNEL_IP=192.168.33. 183 THISHOST_TUNNEL_NETMASK=24 juno-controller:/root/env.sh juno-network:/root/env.sh juno-compute01/02:/root/env.sh All node 192.168.X.184 11. 11 Env setting IP setting #!/bin/bash #get primary NIC info for i in $(ls /sys/class/net); do NIC=$i MY_MAC=$(cat /sys/class/net/$i/address) if [ "$(cat /sys/class/net/$i/ifindex)" == '2' ]; then #setup the IP configuration for management NIC sed -i.bak "s/dhcp/none/g" /etc/sysconfig/network-scripts/ifcfg-$NIC sed -i "s/HWADDR/#HWADDR/g" /etc/sysconfig/network-scripts/ifcfg-$NIC sed -i "/#HWADDR/a HWADDR="$MY_MAC"" /etc/sysconfig/network-scripts/ifcfg-$NIC sed -i "s/UUID/#UUID/g" /etc/sysconfig/network-scripts/ifcfg-$NIC echo "IPADDR="$THISHOST_IP"" >> /etc/sysconfig/network-scripts/ifcfg-$NIC echo "PREFIX="$THISHOST_NETMASK"" >> /etc/sysconfig/network-scripts/ifcfg-$NIC echo "GATEWAY="$THISHOST_GATEWAY"" >> /etc/sysconfig/network-scripts/ifcfg-$NIC echo "DNS1="$THISHOST_DNS"" >> /etc/sysconfig/network-scripts/ifcfg-$NIC mv /etc/sysconfig/network-scripts/ifcfg-$NIC.bak . fi if [ "$(cat /sys/class/net/$i/ifindex)" == '3' ]; then #create config file for Tunnel NIC echo "HWADDR="$MY_MAC"" > /etc/sysconfig/network-scripts/ifcfg-$NIC echo "TYPE="Ethernet"" >> /etc/sysconfig/network-scripts/ifcfg-$NIC echo "BOOTPROTO="none"" >> /etc/sysconfig/network-scripts/ifcfg-$NIC echo "IPV4_FAILURE_FATAL="no"" >> /etc/sysconfig/network-scripts/ifcfg-$NIC echo "NAME="$NIC"" >> /etc/sysconfig/network-scripts/ifcfg-$NIC echo "ONBOOT="yes"" >> /etc/sysconfig/network-scripts/ifcfg-$NIC echo "IPADDR="$THISHOST_TUNNEL_IP"" >> /etc/sysconfig/network-scripts/ifcfg-$NIC echo "PREFIX="$THISHOST_TUNNEL_NETMASK"" >> /etc/sysconfig/network-scripts/ifcfg-$NIC fi if [ "$(cat /sys/class/net/$i/ifindex)" == '4' ]; then #create config file for External NIC echo "HWADDR="$MY_MAC"" > /etc/sysconfig/network-scripts/ifcfg-$NIC echo "TYPE="Ethernet"" >> /etc/sysconfig/network-scripts/ifcfg-$NIC echo "BOOTPROTO="none"" >> /etc/sysconfig/network-scripts/ifcfg-$NIC echo "IPV4_FAILURE_FATAL="no"" >> /etc/sysconfig/network-scripts/ifcfg-$NIC echo "NAME="$NIC"" >> /etc/sysconfig/network-scripts/ifcfg-$NIC echo "ONBOOT="yes"" >> /etc/sysconfig/network-scripts/ifcfg-$NIC fi done 먼저 각 node에서 env.sh를 실행한다. 그리고 아래대로 network-sciprt를 실행한다. All node #setup hostname cp -f /dev/null /etc/hostname echo " $THISHOST_NAME " > /etc/hostname echo "$THISHOST_IP $THISHOST_NAME" >> /etc/hosts 12. 12 CONTROLLER NODE SETTING Openstack DB setting yum -y install mariadb mariadb-server MySQL-python controller node = controller + MariaDB,RabbitMQ,Glance,NOVA api/scheduler,Neutron api, cinder api egrep -v "^#|^$" /etc/my.cnf [mysqld] bind-address = 192.168.32.181 default-storage-engine = innodb innodb_file_per_table collation-server = utf8_general_ci init-connect = 'SET NAMES utf8' character-set-server = utf8 sed -i.bak "10i bind-address = $CONTROLLER_IPn default-storage-engine = innodbn innodb_file_per_tablen collation-server = utf8_general_cin init-connect = 'SET NAMES utf8'n character-set-server = utf8n " /etc/my.cnf #edit /etc/my.cnf systemctl enable mariadb.service systemctl start mariadb.service mysql_secure_installation # mariadb 암호 설정 Enter current password for root (enter for none): Enter OK, successfully used password, moving on... Setting the root password ensures that nobody can log into the MariaDB root user without the proper authorisation. Set root password? [Y/n] Y New password: password ; Re-enter new password: password Password updated successfully! Reloading privilege tables.. ... Success! # Enter *3 13. 13 CONTROLLER NODE SETTING Openstack DB 생성 mysql -u root -p > keystonerc_admin echo "export OS_AUTH_URL=http://$CONTROLLER_IP:35357/v2.0">> keystonerc_admin source keystonerc_admin 아래 쉘을 모든 노드에 등록하여, admin이 항상 관리할 수 있도록 한다. 보통 .bashrc에 등록하여 사용함. All node keystone user-list keystone user-role-list keystone tenant-list keystone token-get 17. 17 CONTROLLER NODE SETTING glance-api / glance-registry / Database / Storage repository for image files Glance는 인스턴스의 운영제제 이미지 파일을 관리한다. yum -y install openstack-glance python-glanceclient [DEFAULT] [database] connection = mysql://glance:service@192.168.32.181/glance [keystone_authtoken] auth_uri = http://192.168.32.181:5000/v2.0 identity_uri = http://192.168.32.181:35357 admin_tenant_name = service admin_user = glance admin_password = service [paste_deploy] flavor = keystone [glance_store] default_store = file filesystem_store_datadir=/var/lib/glance/im ages/ # Cluster native mount point sed -i.bak "/[database]/a connection = mysql://glance:$SERVICE_PWD@$CONTROLLER_IP/glance" /etc/glance/glance-api.conf sed -i "/[keystone_authtoken]/a auth_uri = http://$CONTROLLER_IP:5000/v2.0n identity_uri = http://$CONTROLLER_IP:35357n admin_tenant_name = servicen admin_user = glancen admin_password = $SERVICE_PWD" /etc/glance/glance- api.conf sed -i "/[paste_deploy]/a flavor = keystone" /etc/glance/glance-api.conf sed -i "/[glance_store]/a default_store = filen filesystem_store_datadir = /var/lib/glance/images/" /etc/glance/glance-api.conf #edit /etc/glance/glance-api.conf keystone user-create --name glance --pass $SERVICE_PWD keystone user-role-add --user glance --tenant service --role admin keystone service-create --name glance --type image --description "OpenStack Image Service" keystone endpoint-create --service-id $(keystone service-list | awk '/ image / {print $2}') --publicurl http://$CONTROLLER_IP:9292 --internalurl http://$CONTROLLER_IP:9292 --adminurl http://$CONTROLLER_IP:9292 --region regionOne egrep -v "^#|^$" /etc/glance/glance-api.conf 18. 18 CONTROLLER NODE SETTING glance-api / glance-registry / Database / Storage repository for image files 이미지 등록을 담당하는 glance-registry 세팅 / [DEFAULT] [database] connection = mysql://glance:service@192.168.32.181/glance [keystone_authtoken] auth_uri = http://192.168.32.181:5000/v2.0 identity_uri = http://192.168.32.181:35357 admin_tenant_name = service admin_user = glance admin_password = service [paste_deploy] flavor = keystone [profiler] sed -i.bak "/[database]/a connection = mysql://glance:$SERVICE_PWD@$CONTROLLER_IP/glance" /etc/glance/glance-registry.conf sed -i "/[keystone_authtoken]/a auth_uri = http://$CONTROLLER_IP:5000/v2.0n identity_uri = http://$CONTROLLER_IP:35357n admin_tenant_name = servicen admin_user = glancen admin_password = $SERVICE_PWD" /etc/glance/glance- registry.conf sed -i "/[paste_deploy]/a flavor = keystone" /etc/glance/glance-registry.conf #edit /etc/glance/glance-registry.conf #start glance su -s /bin/sh -c "glance-manage db_sync" glance systemctl enable openstack-glance-api.service openstack-glance-registry.service systemctl start openstack-glance-api.service openstack-glance-registry.service #upload the cirros image to glance yum -y install wget wget http://cdn.download.cirros-cloud.net/0.3.3/cirros-0.3.3-x86_64-disk.img glance image-create --name "cirros-0.3.3-x86_64" --file cirros-0.3.3-x86_64-disk.img --disk-format qcow2 --container-format bare --is-public True --progress glance image-create --name 'centos7' --disk-format qcow2 --container-format bare --is-public true --copy- from http://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud-20141129_01.qcow2 glance image-list egrep -v "^#|^$" /etc/glance/glance-registry.conf 19. 19 CONTROLLER NODE SETTING nova-api / nova-compute /nova-scheduler /nova-conductor module 인스턴스 생성 및 삭제를 책임지는 nova 설치 #install the nova controller components (To install and configure Compute controller components) yum -y install openstack-nova-api openstack-nova-cert openstack-nova-conductor openstack-nova-console openstack-nova-novncproxy openstack-nova-scheduler python-novaclient keystone user-create --name nova --pass $SERVICE_PWD keystone user-role-add --user nova --tenant service --role admin keystone service-create --name nova --type compute --description "OpenStack Compute" keystone endpoint-create --service-id $(keystone service-list | awk '/ compute / {print $2}') --publicurl http://$CONTROLLER_IP:8774/v2/%(tenant_id)s --internalurl http://$CONTROLLER_IP:8774/v2/%(tenant_id)s --adminurl http://$CONTROLLER_IP:8774/v2/%(tenant_id)s --region regionOne 20. 20 CONTROLLER NODE SETTING nova-api / nova-compute /nova-scheduler /nova-conductor module Nova.conf 파일 구성 / egrep -v "^#|^$" /etc/nova/nova.conf [DEFAULT] rpc_backend = rabbit rabbit_host = 192.168.32.181 auth_strategy = keystone my_ip = 192.168.32.181 vncserver_listen = 192.168.32.181 vncserver_proxyclient_address = 192.168.32.181 network_api_class = nova.network.neutronv2.api.API security_group_api = neutron linuxnet_interface_driver = nova.network.linux_net.LinuxOVSInterfaceDriver firewall_driver = nova.virt.firewall.NoopFirewallDriver [baremetal] [glance] host = 192.168.32.181 [hyperv] [database] connection = mysql://nova:service@192.168.32.181/nova [keystone_authtoken] auth_uri = http://192.168.32.181:5000/v2.0 identity_uri = http://192.168.32.181:35357 admin_tenant_name = service admin_user = nova admin_password = service [neutron] url = http://192.168.32.181:9696 auth_strategy = keystone admin_auth_url = http://192.168.32.181:35357/v2.0 admin_tenant_name = service admin_username = neutron admin_password = service service_metadata_proxy = True metadata_proxy_shared_secret = meta123 sed -i.bak "/[database]/a connection = mysql://nova:$SERVICE_PWD@$CONTROLLER_IP/nova" /etc/nova/nova.conf sed -i "/[DEFAULT]/a rpc_backend = rabbitn rabbit_host = $CONTROLLER_IPn auth_strategy = keystonen my_ip = $CONTROLLER_IPn vncserver_listen = $CONTROLLER_IPn vncserver_proxyclient_address = $CONTROLLER_IPn network_api_class = nova.network.neutronv2.api.APIn security_group_api = neutronn linuxnet_interface_driver = nova.network.linux_net.LinuxOVSInterfaceDrivern firewall_driver = nova.virt.firewall.NoopFirewallDriver" /etc/nova/nova.conf sed -i "/[keystone_authtoken]/i [database]nconnection = mysql://nova:$SERVICE_PWD@$CONTROLLER_IP/nova" /etc/nova/nova.conf sed -i "/[keystone_authtoken]/a auth_uri = http://$CONTROLLER_IP:5000/v2.0n identity_uri = http://$CONTROLLER_IP:35357n admin_tenant_name = servicen admin_user = novan admin_password = $SERVICE_PWD" /etc/nova/nova.conf sed -i "/[glance]/a host = $CONTROLLER_IP" /etc/nova/nova.conf sed -i "/[neutron]/a url = http://$CONTROLLER_IP:9696n auth_strategy = keystonen admin_auth_url = http://$CONTROLLER_IP:35357/v2.0n admin_tenant_name = servicen admin_username = neutronn admin_password = $SERVICE_PWDn service_metadata_proxy = Truen metadata_proxy_shared_secret = $META_PWD" /etc/nova/nova.conf #edit /etc/nova/nova.conf 21. 21 CONTROLLER NODE SETTING nova-api / nova-compute /nova-scheduler /nova-conductor module nova 서비스 구동 #start nova su -s /bin/sh -c "nova-manage db sync" nova systemctl enable openstack-nova-api.service openstack-nova-cert.service openstack-nova-consoleauth.service openstack-nova-scheduler.service openstack-nova-conductor.service openstack-nova-novncproxy.service systemctl start openstack-nova-api.service openstack-nova-cert.service openstack-nova-scheduler.service openstack-nova-conductor.service openstack-nova-novncproxy.service openstack-neutron /openstack-neutron-ml2 / python-neutronclient Neutron 서버 설치 #create keystone entries for neutron keystone user-create --name neutron --pass $SERVICE_PWD keystone user-role-add --user neutron --tenant service --role admin keystone service-create --name neutron --type network --description "OpenStack Networking" keystone endpoint-create --service-id $(keystone service-list | awk '/ network / {print $2}') --publicurl http://$CONTROLLER_IP:9696 --internalurl http://$CONTROLLER_IP:9696 --adminurl http://$CONTROLLER_IP:9696 --region regionOne #install neutron yum -y install openstack-neutron openstack-neutron-ml2 python-neutronclient 22. 22 CONTROLLER NODE SETTING openstack-neutron /openstack-neutron-ml2 / python-neutronclient Neutron 서버를 설치함 / egrep -v "^#|^$" /etc/neutron/neutron.conf [DEFAULT] rpc_backend = rabbit rabbit_host = 192.168.32.181 auth_strategy = keystone core_plugin = ml2 service_plugins = router allow_overlapping_ips = True notify_nova_on_port_status_changes = True notify_nova_on_port_data_changes = True nova_url = http://192.168.32.181:8774/v2 nova_admin_auth_url = http://192.168.32.181:35357/v2.0 nova_region_name = regionOne nova_admin_username = nova nova_admin_tenant_id = 2ec220d040994c4589fb60afc98fc5c3 nova_admin_password = service [matchmaker_redis] [matchmaker_ring] [quotas] [agent] [keystone_authtoken] auth_uri = http://192.168.32.181:5000/v2.0 identity_uri = http://192.168.32.181:35357 admin_tenant_name = service admin_user = neutron admin_password = service [database] connection = mysql://neutron:service@192.168.32.181/neutron connection = mysql://neutron:service@192.168.32.181/neutron connection = mysql://neutron:service@192.168.32.181/neutron sed -i.bak "/[database]/a connection = mysql://neutron:$SERVICE_PWD@$CONTROLLER_IP/neutron" /etc/neutron/neutron.conf SERVICE_TENANT_ID=$(keystone tenant-list | awk '/ service / {print $2}') sed -i '0,/[DEFAULT]/s//[DEFAULT] rpc_backend = rabbit rabbit_host = '"$CONTROLLER_IP"' auth_strategy = keystone core_plugin = ml2 service_plugins = router allow_overlapping_ips = True notify_nova_on_port_status_changes = True notify_nova_on_port_data_changes = True nova_url = http://'"$CONTROLLER_IP"':8774/v2 nova_admin_auth_url = http://'"$CONTROLLER_IP"':35357/v2.0 nova_region_name = regionOne nova_admin_username = nova nova_admin_tenant_id = '"$SERVICE_TENANT_ID"' nova_admin_password = '"$SERVICE_PWD"'/' /etc/neutron/neutron.conf sed -i "/[keystone_authtoken]/a auth_uri = http://$CONTROLLER_IP:5000/v2.0n identity_uri = http://$CONTROLLER_IP:35357n admin_tenant_name = servicen admin_user = neutronn admin_password = $SERVICE_PWD" /etc/neutron/neutron.conf #edit /etc/neutron/neutron.conf 23. 23 CONTROLLER NODE SETTING openstack-neutron /openstack-neutron-ml2 / python-neutronclient Neutron 기본 plug-in인 ML2 사용 / GRE/Openvswitch 사용 / egrep -v "^#|^$" /etc/neutron/plugins/ml2/ml2_conf.ini [ml2] type_drivers = flat,gre tenant_network_types = gre mechanism_drivers = openvswitch [ml2_type_gre] tunnel_id_ranges = 1:1000 [securitygroup] enable_security_group = True enable_ipset = True firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridIptablesFire wallDriver #edit /etc/neutron/plugins/ml2/ml2_conf.ini sed -i "/[ml2]/a type_drivers = flat,gren tenant_network_types = gren mechanism_drivers = openvswitch" /etc/neutron/plugins/ml2/ml2_conf.ini sed -i "/[ml2_type_gre]/a tunnel_id_ranges = 1:1000" /etc/neutron/plugins/ml2/ml2_conf.ini sed -i "/[securitygroup]/a enable_security_group = Truen enable_ipset = Truen firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridIptablesFir ewallDriver" /etc/neutron/plugins/ml2/ml2_conf.ini #edit /etc/neutron/plugins/ml2/ml2_conf.ini #start neutron ln -s /etc/neutron/plugins/ml2/ml2_conf.ini /etc/neutron/plugin.ini su -s /bin/sh -c "neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini upgrade juno" neutron systemctl restart openstack-nova-api.service openstack-nova-scheduler.service openstack-nova-conductor.service systemctl enable neutron-server.service systemctl start neutron-server.service 24. 24 CONTROLLER NODE SETTING openstack-dashboard httpd mod_wsgi memcached python-memcached 웹기반 인터페이스인 dashboard 설치 ALLOWED_HOSTS = ['*'] OPENSTACK_HOST = "192.168.32.181" sed -i.bak "s/ALLOWED_HOSTS = ['horizon.example.com', 'localhost']/ALLOWED_HOSTS = ['*']/" /etc/openstack- dashboard/local_settings sed -i 's/OPENSTACK_HOST = "127.0.0.1"/OPENSTACK_HOST = "'"$CONTROLLER_IP"'"/' /etc/openstack- dashboard/local_settings #edit /etc/openstack-dashboard/local_settings yum -y install openstack-dashboard httpd mod_wsgi memcached python-memcached #start dashboard setsebool -P httpd_can_network_connect on chown -R apache:apache /usr/share/openstack-dashboard/static systemctl enable httpd.service memcached.service systemctl start httpd.service memcached.service egrep -v "^#|^$" /etc/openstack-dashboard/local_settings 25. 25 CONTROLLER NODE SETTING openstack-cinder/ python-cinderclient / python-oslo-db 인스턴스 블록스토리지를 관제를 cinder controller 설치 #create keystone entries for cinder keystone user-create --name cinder --pass $SERVICE_PWD keystone user-role-add --user cinder --tenant service --role admin keystone service-create --name cinder --type volume --description "OpenStack Block Storage" keystone service-create --name cinderv2 --type volumev2 --description "OpenStack Block Storage" keystone endpoint-create --service-id $(keystone service-list | awk '/ volume / {print $2}') --publicurl http://$CONTROLLER_IP:8776/v1/%(tenant_id)s --internalurl http://$CONTROLLER_IP:8776/v1/%(tenant_id)s --adminurl http://$CONTROLLER_IP:8776/v1/%(tenant_id)s --region regionOne keystone endpoint-create --service-id $(keystone service-list | awk '/ volumev2 / {print $2}') --publicurl http://$CONTROLLER_IP:8776/v2/%(tenant_id)s --internalurl http://$CONTROLLER_IP:8776/v2/%(tenant_id)s --adminurl http://$CONTROLLER_IP:8776/v2/%(tenant_id)s --region regionOne #install cinder controller yum -y install openstack-cinder python-cinderclient python-oslo-db 26. 26 CONTROLLER NODE SETTING openstack-cinder/ python-cinderclient / python-oslo-db cinder controller 설치 / egrep -v "^#|^$" /etc/cinder/cinder.conf [DEFAULT] rpc_backend = rabbit rabbit_host = 192.168.32.181 auth_strategy = keystone my_ip = 192.168.32.181 [database] connection = mysql://cinder:service@192.168.32.181/cinder [keystone_authtoken] auth_uri = http://192.168.32.181:5000/v2.0 identity_uri = http://192.168.32.181:35357 admin_tenant_name = service admin_user = cinder admin_password = service sed -i.bak "/[database]/a connection = mysql://cinder:$SERVICE_PWD@$CONTROLLER_IP/cinde r" /etc/cinder/cinder.conf sed -i "0,/[DEFAULT]/a rpc_backend = rabbitn rabbit_host = $CONTROLLER_IPn auth_strategy = keystonen my_ip = $CONTROLLER_IP" /etc/cinder/cinder.conf sed -i "/[keystone_authtoken]/a auth_uri = http://$CONTROLLER_IP:5000/v2.0n identity_uri = http://$CONTROLLER_IP:35357n admin_tenant_name = servicen admin_user = cindern admin_password = $SERVICE_PWD" /etc/cinder/cinder.conf #edit /etc/cinder/cinder.conf egrep -v "^#|^$" /etc/cinder/cinder.conf 27. 27 CONTROLLER NODE SETTING openstack-cinder/ python-cinderclient / python-oslo-db cinder controller start #start cinder controller su -s /bin/sh -c "cinder-manage db sync" cinder systemctl enable openstack-cinder-api.service openstack-cinder-scheduler.service systemctl start openstack-cinder-api.service openstack-cinder-scheduler.service 28. 28 NETWORK NODE openstack-neutron neutron-ml2 neutron-openvswitch echo 'net.ipv4.ip_forward=1' >> /etc/sysctl.conf echo 'net.ipv4.conf.all.rp_filter=0' >> /etc/sysctl.conf echo 'net.ipv4.conf.default.rp_filter=0' >> /etc/sysctl.conf sysctl -p 29. 29 NETWORK NODE openstack-neutron neutron-ml2 neutron-openvswitch Controller에 openvswitch와 L3 부분이 추가된다고 생각하면 된다 . 다른 부분은 진한 글씨체임 egrep -v "^#|^$" /etc/neutron/neutron.conf / egrep -v "^#|^$" /etc/neutron/l3_agent.ini yum -y install openstack-neutron openstack-neutron-ml2 openstack-neutron-openvswitch [DEFAULT] rpc_backend = rabbit rabbit_host = 192.168.0.181 auth_strategy = keystone core_plugin = ml2 service_plugins = router allow_overlapping_ips = True [matchmaker_redis] [keystone_authtoken] auth_uri = http://192.168.0.181:5000/v2.0 identity_uri = http://192.168.0.181:35357 admin_tenant_name = service admin_user = neutron admin_password = service sed -i '0,/[DEFAULT]/s//[DEFAULT] rpc_backend = rabbit rabbit_host = '"$CONTROLLER_IP"' auth_strategy = keystone core_plugin = ml2 service_plugins = router allow_overlapping_ips = True/' /etc/neutron/neutron.conf sed -i "/[keystone_authtoken]/a auth_uri = http://$CONTROLLER_IP:5000/v2.0n identity_uri = http://$CONTROLLER_IP:35357n admin_tenant_name = servicen admin_user = neutronn admin_password = $SERVICE_PWD" /etc/neutron/neutron.conf #edit /etc/neutron/neutron.conf [DEFAULT] interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver use_namespaces = True external_network_bridge = br-ex sed -i "/[DEFAULT]/a interface_driver = neutron.agent.linux.interface.OVSInterfaceDrivern use_namespaces = Truen external_network_bridge = br-ex" /etc/neutron/l3_agent.ini #edit /etc/neutron/l3_agent.ini 30. 30 NETWORK NODE openstack-neutron neutron-ml2 neutron-openvswitch Openvswitch setting / egrep -v "^#|^$" /etc/neutron/plugins/ml2/ml2_conf.ini [ml2] type_drivers = flat,gre tenant_network_types = gre mechanism_drivers = openvswitch [ml2_type_flat] flat_networks = external [ml2_type_vlan] [ml2_type_gre] tunnel_id_ranges = 1:1000 [ml2_type_vxlan] [securitygroup] enable_security_group = True enable_ipset = True firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridIp tablesFirewallDriver [ovs] local_ip = 192.168.33.182 enable_tunneling = True bridge_mappings = external:br-ex [agent] tunnel_types = gre sed -i "/[ml2]/a type_drivers = flat,gren tenant_network_types = gren mechanism_drivers = openvswitch" /etc/neutron/plugins/ml2/ml2_conf.ini sed -i "/[ml2_type_flat]/a flat_networks = external" /etc/neutron/plugins/ml2/ml2_conf.ini sed -i "/[ml2_type_gre]/a tunnel_id_ranges = 1:1000" /etc/neutron/plugins/ml2/ml2_conf.ini sed -i "/[securitygroup]/a enable_security_group = Truen enable_ipset = Truen firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridI ptablesFirewallDrivern [ovs]n local_ip = $THISHOST_TUNNEL_IPn enable_tunneling = Truen bridge_mappings = external:br-exn [agent]n tunnel_types = gre" /etc/neutron/plugins/ml2/ml2_conf.ini #edit /etc/neutron/plugins/ml2/ml2_conf.ini 31. 31 NETWORK NODE openstack-neutron neutron-ml2 neutron-openvswitch Openvswitch setting egrep -v "^#|^$" / etc/neutron/dhcp_agent.ini [root@juno-network neutron]# egrep -v "^#|^$" /etc/neutron/l3_agent.ini [DEFAULT] interface_driver = neutron.agent.linux.interface.OVSInterface Driver use_namespaces = True external_network_bridge = br-ex interface_driver = neutron.agent.linux.interface.OVSInterface Driver use_namespaces = True external_network_bridge = br-ex sed -i "/[DEFAULT]/a interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver n use_namespaces = Truen external_network_bridge = br-ex" /etc/neutron/l3_agent.ini sed -i "/[DEFAULT]/a interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver n dhcp_driver = neutron.agent.linux.dhcp.Dnsmasqn use_namespaces = True" /etc/neutron/dhcp_agent.ini #edit /etc/neutron/dhcp_agent.ini 32. 32 NETWORK NODE openstack-neutron neutron-ml2 neutron-openvswitch metadata_agent 정보 입력 [DEFAULT] auth_url = http://192.168.32.181:5000/v2.0 auth_region = regionOne admin_tenant_name = service admin_user = neutron admin_password = service nova_metadata_ip = 192.168.32.181 metadata_proxy_shared_secret = meta123 sed -i "s/auth_url/#auth_url/g" /etc/neutron/metadata_agent.ini sed -i "s/auth_region/#auth_region/g" /etc/neutron/metadata_agent.ini sed -i "s/admin_tenant_name/#admin_tenant_name/g" /etc/neutron/metadata_agent.ini sed -i "s/admin_user/#admin_user/g" /etc/neutron/metadata_agent.ini sed -i "s/admin_password/#admin_password/g" /etc/neutron/metadata_agent.ini sed -i "/[DEFAULT]/a auth_url = http://$CONTROLLER_IP:5000/v2.0n auth_region = regionOnen admin_tenant_name = servicen admin_user = neutronn admin_password = $SERVICE_PWDn nova_metadata_ip = $CONTROLLER_IPn metadata_proxy_shared_secret = $META_PWD" /etc/neutron/metadata_agent.ini #edit /etc/neutron/metadata_agent.ini egrep -v "^#|^$" /etc/neutron/metadata_agent.ini 33. 33 NETWORK NODE openstack-neutron neutron-ml2 neutron-openvswitch NIC 외부네트웍 연결 #get external NIC info for i in $(ls /sys/class/net); do if [ "$(cat /sys/class/net/$i/ifindex)" == '4' ]; then NIC=$i MY_MAC=$(cat /sys/class/net/$i/address) echo "$i ($MY_MAC)" fi done systemctl enable openvswitch.service systemctl start openvswitch.service ovs-vsctl add-br br-ex ovs-vsctl add-port br-ex $NIC ethtool -K $NIC gro off 34. 34 NETWORK NODE openstack-neutron neutron-ml2 neutron-openvswitch 외부 nework setting [root@net01 network-scripts]# cat ifcfg-eth2 #HWADDR=00:19:99:D5:AA:D0 TYPE=OVSPort DEVICETYPE=ovs OVS_BRIDGE=br-ex BOOTPROTO=none NAME=eth2 #UUID=33d13b63-9eba-4414-996a-75391a71fc6a DEVICE=eth2 ONBOOT=yes [root@net01 network-scripts]# cat ifcfg-br-ex TYPE=OVSIntPort OVS_BRIDGE=br-ex DEVICETYPE=ovs BOOTPROTO=none IPADDR0=192.168.0.182 PREFIX0=24 DEFROUTE=yes IPV4_FAILURE_FATAL=no IPV6INIT=yes IPV6_AUTOCONF=no IPV6_DEFROUTE=yes IPV6_FAILURE_FATAL=no NAME=br-ex #UUID=33d13b63-9eba-4414-996a-75391a71fc6a DEVICE=br-ex ONBOOT=yes GATEWAY=192.168.0.1 DNS1=8.8.8.8 ln -s /etc/neutron/plugins/ml2/ml2_conf.ini /etc/neutron/plugin.ini cp /usr/lib/systemd/system/neutron-openvswitch-agent.service /usr/lib/systemd/system/neutron-openvswitch-agent.service.orig sed -i 's,plugins/openvswitch/ovs_neutron_plugin.ini,plugin.ini,g' /usr/lib/systemd/system/neutron-openvswitch-agent.service 35. 35 NETWORK NODE 서비스 활성화 systemctl enable neutron-openvswitch-agent.service neutron-l3-agent.service neutron-dhcp-agent.service neutron-metadata-agent.service neutron-ovs-cleanup.service systemctl start neutron-openvswitch-agent.service neutron-l3-agent.service neutron-dhcp-agent.service neutron-metadata-agent.service openstack-neutron neutron-ml2 neutron-openvswitch 36. 36 COMPUTE NODE nova-compute sysfsutils libvirt-daemon-config-nwfilter Compute 클라우드 컴퓨팅을 위한 nova 설치 echo 0 > /sys/fs/selinux/enforce echo 'net.ipv4.conf.all.rp_filter=0' >> /etc/sysctl.conf echo 'net.ipv4.conf.default.rp_filter=0' >> /etc/sysctl.conf sysctl -p yum -y install openstack-nova-compute sysfsutils libvirt-daemon-config-nwfilter 37. 37 COMPUTE NODE nova-compute sysfsutils libvirt-daemon-config-nwfilter /etc/nova/nova.conf / egrep -v "^#|^$" /etc/nova/nova.conf [DEFAULT] rpc_backend = rabbit rabbit_host = 192.168.32.181 auth_strategy = keystone my_ip = 192.168.32.183 vnc_enabled = True vncserver_listen = 0.0.0.0 vncserver_proxyclient_address = 192.168.32.183 novncproxy_base_url = http://192.168.32.181:6080/vnc_auto.html network_api_class = nova.network.neutronv2.api.API security_group_api = neutron linuxnet_interface_driver = nova.network.linux_net.LinuxOVSInterfaceDriver firewall_driver = nova.virt.firewall.NoopFirewallDriver [glance] host = 192.168.32.181 [keystone_authtoken] auth_uri = http://192.168.32.181:5000/v2.0 identity_uri = http://192.168.32.181:35357 admin_tenant_name = service admin_user = nova admin_password = service [neutron] url = http://192.168.32.181:9696 auth_strategy = keystone admin_auth_url = http://192.168.32.181:35357/v2.0 admin_tenant_name = service admin_username = neutron admin_password = service sed -i.bak "/[DEFAULT]/a rpc_backend = rabbitn rabbit_host = $CONTROLLER_IPn auth_strategy = keystonen my_ip = $THISHOST_IPn vnc_enabled = Truen vncserver_listen = 0.0.0.0n vncserver_proxyclient_address = $THISHOST_IPn novncproxy_base_url = http://$CONTROLLER_IP:6080/vnc_auto.htmln network_api_class = nova.network.neutronv2.api.APIn security_group_api = neutronn linuxnet_interface_driver = nova.network.linux_net.LinuxOVSInterfaceDrivern firewall_driver = nova.virt.firewall.NoopFirewallDriver" /etc/nova/nova.conf sed -i "/[keystone_authtoken]/a auth_uri = http://$CONTROLLER_IP:5000/v2.0n identity_uri = http://$CONTROLLER_IP:35357n admin_tenant_name = servicen admin_user = novan admin_password = $SERVICE_PWD" /etc/nova/nova.conf sed -i "/[glance]/a host = $CONTROLLER_IP" /etc/nova/nova.conf 38. 38 COMPUTE NODE openstack-neutron-ml2 openstack-neutron-openvswitch neutron seetting / egrep -v "^#|^$" /etc/neutron/neutron.conf #install neutron yum -y install openstack-neutron-ml2 openstack-neutron-openvswitch [DEFAULT] rpc_backend = rabbit rabbit_host = 192.168.32.181 auth_strategy = keystone core_plugin = ml2 service_plugins = router allow_overlapping_ips = True [keystone_authtoken] auth_uri = http://192.168.32.181:5000/v2.0 identity_uri = http://192.168.32.181:35357 admin_tenant_name = service admin_user = neutron admin_password = service sed -i '0,/[DEFAULT]/s//[DEFAULT] rpc_backend = rabbitn rabbit_host = '"$CONTROLLER_IP"' auth_strategy = keystone core_plugin = ml2 service_plugins = router allow_overlapping_ips = True/' /etc/neutron/neutron.conf sed -i "/[keystone_authtoken]/a auth_uri = http://$CONTROLLER_IP:5000/v2.0n identity_uri = http://$CONTROLLER_IP:35357n admin_tenant_name = servicen admin_user = neutronn admin_password = $SERVICE_PWD" /etc/neutron/neutron.conf #edit /etc/neutron/neutron.conf 39. 39 COMPUTE NODE openstack-neutron-ml2 openstack-neutron-openvswitch ml2_conf.ini 수정 / egrep -v "^#|^$" /etc/neutron/plugins/ml2/ml2_conf.ini [ml2] type_drivers = flat,gre tenant_network_types = gre mechanism_drivers = openvswitch [ml2_type_gre] tunnel_id_ranges = 1:1000 [securitygroup] enable_security_group = True enable_ipset = True firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridIp tablesFirewallDriver [ovs] local_ip = 192.168.33.183 enable_tunneling = True [agent] tunnel_types = gre sed -i "/[ml2]/a type_drivers = flat,gren tenant_network_types = gren mechanism_drivers = openvswitch" /etc/neutron/plugins/ml2/ml2_conf.ini sed -i "/[ml2_type_gre]/a tunnel_id_ranges = 1:1000" /etc/neutron/plugins/ml2/ml2_conf.ini sed -i "/[securitygroup]/a enable_security_group = Truen enable_ipset = Truen firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridI ptablesFirewallDrivern [ovs]n local_ip = $THISHOST_TUNNEL_IPn enable_tunneling = Truen [agent]n tunnel_types = gre" /etc/neutron/plugins/ml2/ml2_conf.ini #edit /etc/neutron/plugins/ml2/ml2_conf.ini systemctl enable openvswitch.service systemctl start openvswitch.service 40. 40 COMPUTE NODE nova-compute node 구성 /etc/nova/nova.conf [neutron] url = http://192.168.32.181:9696 auth_strategy = keystone admin_auth_url = http://192.168.32.181:35357/v2.0 admin_tenant_name = service admin_username = neutron admin_password = service sed -i "/[neutron]/a url = http://$CONTROLLER_IP:9696n auth_strategy = keystonen admin_auth_url = http://$CONTROLLER_IP:35357/v2.0n admin_tenant_name = servicen admin_username = neutronn admin_password = $SERVICE_PWD" /etc/nova/nova.conf #edit /etc/nova/nova.conf ln -s /etc/neutron/plugins/ml2/ml2_conf.ini /etc/neutron/plugin.ini cp /usr/lib/systemd/system/neutron-openvswitch-agent.service /usr/lib/systemd/system/neutron-openvswitch-agent.service.orig sed -i 's,plugins/openvswitch/ovs_neutron_plugin.ini,plugin.ini,g' /usr/lib/systemd/system/neutron-openvswitch-agent.service systemctl enable libvirtd.service openstack-nova-compute.service systemctl start libvirtd.service systemctl start openstack-nova-compute.service systemctl enable neutron-openvswitch-agent.service systemctl start neutron-openvswitch-agent.service egrep -v "^#|^$" /etc/nova/nova.conf 41. 41 COMPUTE NODE openstack-cinder targetcli python-oslo-db MySQL-python Cinder 디스크추가 yum -y install openstack-cinder targetcli python-oslo-db MySQL-python #cinder storage node pvcreate /dev/sdb vgcreate cinder-volumes /dev/sdb [DEFAULT] rpc_backend = rabbit rabbit_host = 192.168.32.181 auth_strategy = keystone my_ip = 192.168.32.183 iscsi_helper = lioadm [database] connection = mysql://cinder:service@192.168.32.181/cinder [keystone_authtoken] auth_uri = http://192.168.32.181:5000/v2.0 identity_uri = http://192.168.32.181:35357 admin_tenant_name = service admin_user = cinder admin_password = service sed -i.bak "/[database]/a connection = mysql://cinder:$SERVICE_PWD@$CONTROLLER_IP/cinde r" /etc/cinder/cinder.conf sed -i '0,/[DEFAULT]/s//[DEFAULT] rpc_backend = rabbit rabbit_host = '"$CONTROLLER_IP"' auth_strategy = keystone my_ip = '"$THISHOST_IP"' iscsi_helper = lioadm/' /etc/cinder/cinder.conf sed -i "/[keystone_authtoken]/a auth_uri = http://$CONTROLLER_IP:5000/v2.0n identity_uri = http://$CONTROLLER_IP:35357n admin_tenant_name = servicen admin_user = cindern admin_password = $SERVICE_PWD" /etc/cinder/cinder.conf #edit /etc/cinder/cinder.conf egrep -v "^#|^$" /etc/cinder/cinder.conf 42. 42 COMPUTE NODE openstack-cinder targetcli python-oslo-db MySQL-python Cinder 디스크추가 sed -i 's/filter/#filter/g ' /etc/lvm/lvm.conf sed -i "/devices {/a filter = ["a/sd/","a/sdb/", "r/.*/"] " /etc/lvm/lvm.conf systemctl enable openstack-cinder-volume.service target.service systemctl start openstack-cinder-volume.service target.service egrep -v "^#|^$" /etc/cinder/cinder.conf [root@juno-controller lvm]# grep filter lvm.conf filter = [ "a/sda/", "a/sdb/", "r/.*/"] cinder create --display_name test3 2 [root@juno-compute cinder]# cinder list +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+ | ID | Status | Display Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+ | 35e69e09-015b-472e-a77c-a06f307beb92 | available | test3 | 2 | None | false +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+ [root@juno-compute cinder]# vgs VG #PV #LV #SN Attr VSize VFree centos 1 2 0 wz--n- 19.51g 0 cinder-volumes 1 1 0 wz--n- 50.00g 48.00g [root@juno-compute cinder]# lvs LV VG Attr LSize Pool Origin Data% Move Log Cpy%Sync Convert root centos -wi-ao---- 17.51g swap centos -wi-ao---- 2.00g volume-35e69e09-015b-472e-a77c-a06f307beb92 cinder-volumes -wi-a----- 2.00g 43. 43 Contents 3. docker 구축 기초 [ 30분] 4. 오픈스택에 docker를 연결 [ 30분] 1. Openstack 인프라 구축 (4 node 구성) [ 30분] 2. Openstack 위에 VM 생성 [ 20분 ] 5. Docker로 WEB서비스 구축 [ 15분] 6. 7. Docker로 jenkins 구현 [30분] Openstack 위에 Docker로 WEB서비스 구축 [ 15분] 44. 44 What to do during 20 min. OpenStack vm 생성 기본적인 vm 생성 방법 neturon network 생성 라우터 생성 게이트웨이 설정 내부 인터페이스 추가 인스턴스 생성 http://docs.openstack.org/juno http://behindtheracks.com/category/juno/ 45. 45 Network만들기 Network 생성 / 외부 subnet 생성 [root@juno-controller ~]# Created a new network: +---------------------------+--------------------------------------+ | Field | Value | +---------------------------+--------------------------------------+ | admin_state_up | True | | id | 74cea9a5-434c-4bff-89b7-a1e503b43d39 | | name | ext-net | | provider:network_type | gre | | provider:physical_network | | | provider:segmentation_id | 2 | | router:external | True | | shared | True | | status | ACTIVE | | subnets | | | tenant_id | e7cb7856091d4d839031d79582c93a76 | +---------------------------+--------------------------------------+ neutron net-create ext-net --shared --router:external True neutron subnet-create ext-net --name ext-subnet --allocation-pool start=192.168.0.200,end=192.168.0.220 --disable-dhcp --gateway 192.168.0.1 192.168.0.0/24 Created a new subnet: +-------------------+----------------------------------------------------+ | Field | Value | +-------------------+----------------------------------------------------+ | allocation_pools | {"start": "192.168.0.200", "end": "192.168.0.220"} | | cidr | 192.168.0.0/24 | | dns_nameservers | | | enable_dhcp | False | | gateway_ip | 192.168.0.1 | | host_routes | | | id | d84f7826-ae27-420f-9f1d-da7261c76e0f | | ip_version | 4 | | ipv6_address_mode | | | ipv6_ra_mode | | | name | ext-subnet | | network_id | 74cea9a5-434c-4bff-89b7-a1e503b43d39 | | tenant_id | e7cb7856091d4d839031d79582c93a76 | +-------------------+----------------------------------------------------+ 46. 46 Network만들기 내부 network 생성 / 내부 subnet 생성 Created a new network: +---------------------------+--------------------------------------+ | Field | Value | +---------------------------+--------------------------------------+ | admin_state_up | True | | id | 666c4f98-2a42-46a0-838c-0a82f7335585 | | name | admin-net | | provider:network_type | gre | | provider:physical_network | | | provider:segmentation_id | 1 | | router:external | False | | shared | False | | status | ACTIVE | | subnets | | | tenant_id | e7cb7856091d4d839031d79582c93a76 | +---------------------------+--------------------------------------+ neutron net-create admin-net Created a new subnet: +-------------------+--------------------------------------------+ | Field | Value | +-------------------+--------------------------------------------+ | allocation_pools | {"start": "10.0.1.2", "end": "10.0.1.254"} | | cidr | 10.0.1.0/24 | | dns_nameservers | | | enable_dhcp | True | | gateway_ip | 10.0.1.1 | | host_routes | | | id | 768d888e-9fee-46ac-9c98-bf2ba82d8a44 | | ip_version | 4 | | ipv6_address_mode | | | ipv6_ra_mode | | | name | admin-subnet | | network_id | 666c4f98-2a42-46a0-838c-0a82f7335585 | | tenant_id | e7cb7856091d4d839031d79582c93a76 | +-------------------+--------------------------------------------+ neutron subnet-create admin-net --name admin-subnet --gateway 10.0.1.1 10.0.1.0/24 47. 47 Network만들기 라우터 생성 / 외부라우터 연결 Created a new router: +-----------------------+--------------------------------------+ | Field | Value | +-----------------------+--------------------------------------+ | admin_state_up | True | | distributed | False | | external_gateway_info | | | ha | False | | id | 14e32baa-bab5-4f72-9f19-ebd00f5d3671 | | name | admin-router | | routes | | | status | ACTIVE | | tenant_id | e7cb7856091d4d839031d79582c93a76 | +-----------------------+--------------------------------------+ neutron router-create admin-router Added interface b5913593-5e66-44cb-8d4a-88ae7c803162 to router admin-router.ㅍ Set gateway for router admin-router neutron router-gateway-set admin-router ext-net neutron router-interface-add admin-router admin-subnet 48. 48 Network만들기 Vm생성 neutron router-interface-add admin-router admin-subnet 49. 49 Contents 3. docker 구축 기초 [ 30분] 4. 오픈스택에 docker를 연결 [ 30분] 1. Openstack 인프라 구축 (4 node 구성) [ 30분] 2. Openstack 위에 VM 생성 [ 20분 ] 5. Docker로 WEB서비스 구축 [ 15분] 6. 7. Docker로 jenkins 구현 [30분] Openstack 위에 Docker로 WEB서비스 구축 [ 15분] 50. 50 What to do during 30 min. Docker / Container / Openstack / CI Docker는 개발자와 시스템 관리자가 어플리케이션을 개발하고, 배포하고, 운영하는 플랫폼 Docker는 이미지 기반(애플리케이션과 그 환경까지를 모두 포함) / 그 이미지를 기반으로 컨테이너 형태로 구현 30분 동안 우리가 할 것 (시스템 인스톨 / 애플리케이션 구축 / 구성 / 도커 이미지 관리 / 업로드 ) __ _____| | | __| | ___ _ __ ___ | | / / / _ | | / _` |/ _ | '_ / _ | | V V / __/ | | | (_| | (_) | | | | __/ |_| _/_/ ___|_|_| __,_|___/|_| |_|___| (_) ## . ## ## ## == ## ## ## ## === /""""""""""""""""___/ === ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ / ===- ~~~ ______ o __/ __/ __________/ | | __ | __ __ | _ __ _ / | / / |/ / _ | __/| __/ __ |_ __ | 51. 51 52. 52 53. 53 Getting Started with Docker Hub. Docker Hub 먼저 docker site에 가입 (history 관리) Hub.docker.com [root@Tiger ~]# docker login Username: oscinfra Password: Email: khoj@osci.kr Login Succeeded 54. 54 Installing Docker on CentOS 7 . Hostname setting Docker install / Docker start 등록 / 최신버전 update #env setting export LANG=en_US.UTF-8; #hostname setting yes | cp -f /dev/null /etc/hostname;echo "docker" > /etc/hostname ; hostname; systemctl disable firewalld; systemctl stop firewalld #repo copy scp 192.168.0.220:/etc/yum.repos.d/osc.repo /etc/yum.repos.d/osc.repo #docker installation/setting yum -y install update > /dev/null ; yum -y install docker; systemctl enable docker; systemctl start docker; systemctl status docker | grep Active #update docker to latest version #wget https://get.docker.com/builds/Linux/x86_64/docker-latest -O $(type -P docker) # check the version docker version 55. 55 . Docker 로 할 수 있는 일 Image 가지고 오기 새로운 container 만들기 파일시스템 할당하여, read-wirte layer 마운트 하기 네트웍과 브릿지 인터페이스 할당하기 IP 주소 할당하기 프로세스 실행하기 프로세스/어플리케이션 output 보여주기 [root@juno-compute yum.repos.d]# docker search centos NAME DESCRIPTION STARS OFFICIAL AUTOMATED centos The official build of CentOS. 864 [OK] docker ps docker logs docker stop docker build docker commit docker cp . docker diff Docker command Downloading container images & Running Docker Containers [root@juno-compute ~]# docker images REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE centos 7 88f9454e60dd 4 days ago 223.9 MB centos centos7 88f9454e60dd 4 days ago 223.9 MB centos latest 88f9454e60dd 4 days ago 223.9 MB [root@juno-compute ~]# docker pull centos Pulling repository centos 88f9454e60dd: Download complete Status: Image is up to date for centos:latest [root@juno-compute ~]# docker run centos /bin/echo "Hello World" Hello World 56. 56 . [root@docker /]# docker info Containers: 6 Images: 18 Storage Driver: devicemapper Pool Name: docker-253:1-67220484-pool Pool Blocksize: 65.54 kB Data file: /var/lib/docker/devicemapper/devicemappe r/data Metadata file: /var/lib/docker/devicemapper/devicemappe r/metadata Data Space Used: 831.9 MB Data Space Total: 107.4 GB Metadata Space Used: 1.696 MB Metadata Space Total: 2.147 GB Library Version: 1.02.84-RHEL7 (2014- 03-26) Execution Driver: native-0.2 Kernel Version: 3.10.0- 123.20.1.el7.x86_64 Operating System: CentOS Linux 7 (Core) Downloading container images & Running Docker Containers [root@docker ~]# docker info Containers: 15 Images: 43 Storage Driver: devicemapper Pool Name: docker-253:1-67220484-pool Pool Blocksize: 65.54 kB Data file: /var/lib/docker/devicemapper/devicemapper/ data Metadata file: /var/lib/docker/devicemapper/devicemapper/ metadata Data Space Used: 2.682 GB Data Space Total: 107.4 GB Metadata Space Used: 4.174 MB Metadata Space Total: 2.147 GB Library Version: 1.02.84-RHEL7 (2014-03- 26) Execution Driver: native-0.2 Kernel Version: 3.10.0-123.20.1.el7.x86_64 Operating System: CentOS Linux 7 (Core) Username: oscinfra Registry: [https://index.docker.io/v1/] Docker info 57. 57 . : A "Hello worldâ / An Interactive Container 컨테이너를 상자 안의 프로세스라고 비유할 수 있습니다. 상자는 프로세스가 필요로 하는 모든 것, 파일 시스템, 시스템 라이브러리, 쉘 등을 가지고 있지만, 기본적으로 아무것도 실행하고 있지 않습니다. 그 프로세스들을 실행함으로써, 컨테이너를 시작합니다. [root@docker ~]# docker run centos echo "Hellow world" Hellow world Run your new image [ Dockerizing Applications ] [root@docker01 ~]# docker run -i -t --name osc centos /bin/bash # i,-- interacitve . -t,--tty [root@6cfe2306796b /]# ip a | grep eth0 11: eth0: mtu 1500 qdisc pfifo_fast state UP qlen 1000 inet 172.17.0.4/16 scope global eth0 [root@6cfe2306796b /]# yum install -y wget > /dev/null ; exit [root@docker ~]# docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 6cfe2306796b centos:7 "/bin/bash" 5 minutes ago Exited (0) 46 seconds ago osc [root@docker ~]# docker start osc ;docker ps ; docker attach osc osc [root@6cfe2306796b /]# rpm -qa wget wget-1.14-10.el7_0.1.x86_64 [root@6cfe2306796b /]# ctrl+p , ctrl+q # shell 종료없이 호스트로 복귀 출처 : https://docs.docker.com/userguide/usingdocker/ 58. 58 . A Daemonized Hello world 컨테이터를 데몬처럼 돌리기 [root@docker ~]# docker images REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE centos 7 88f9454e60dd 2 days ago 223.9 MB centos centos7 88f9454e60dd 2 days ago 223.9 MB centos latest 88f9454e60dd 2 days ago 223.9 MB [root@docker ~]# docker run --name osc -d centos /bin/bash -c "while true;do echo Hello world;sleep 1;done" 61fe6bea107205c3ba9bfe998e506297797f0491d6bbe32664f5db261641c5ee [root@docker01 ~]# ps -ef | grep âv grep | grep true root 5338 3689 0 16:18 ? 00:00:00 /bin/bash -c while true;do echo Hello world;sleep 1;done root 5357 3525 0 16:18 pts/0 00:00:00 grep --color=auto true [root@docker ~]# docker logs --tail=10 -ft osc 2015-03-07T11:07:18.655548350Z Hello world [root@docker ~]# docker top osc UID PID PPID C STIME TTY TIME CMD root 3408 983 0 06:02 ? 00:00:00 /bin/bash -c while true;do echo Hello world;sleep 1;done root 3827 3408 0 06:08 ? 00:00:00 sleep 1 [root@docker ~]# docker inspect osc / docker stop osc / docker kill osc Run your new image [ Dockerizing Applications ] 59. 59 . We met some errors as below / where is the docker definition file? [root@docker01 ~]# docker run -i -t --name test learn/tutorial /bin/bash FATA[0000] Error response from daemon: Conflict. The name "test" is already in use by container c4803f88e5c4. You have to delete (or rename) that container to be able to reuse that name. Think/Think/Think [root@docker01 ~]# docker rm c4803f88e5c4 c4803f88e5c4 [root@docker01 ~]# docker run -i -t --name test learn/tutorial /bin/bash root@1dbb3af8ec20:/# Where is the docker definition file? [root@docker ~]# docker inspect osc # docker inspect -f '{{.Name}}' [{ "AppArmorProfile": "", "Args": [ "-c", "while true;do echo Hello world;sleep 1;doneâ ] #images /var/lib/docker/conatiner [root@docker docker]# cd conatainers/61fe6bea107205c3ba9bfe9* 61fe6bea107205c3ba9bfe998e506297797f0491d6bbe32664f5db261641c5ee]# ls 61feXX-json.log config.json hostconfig.json hostname hosts resolv.conf secrets 60. 60 . training/webapp Running a Web Application in Docker [root@docker01 ~]# docker run -d -P training/webapp python app.py # -d, -- detach=false / [root@docker01 ~]# docker ps -l CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 489160dbba6b training/webapp:latest "python app.py" 5 minutes ago Up 5 minutes 0.0.0.0:49154->5000/tcp serene_heisenberg [root@docker01 ~]# docker logs -f cocky_hopper * Running on http://0.0.0.0:5000/ 192.168.0.4 - - [28/Feb/2015 22:14:54] "GET / HTTP/1.1" 200 - 192.168.0.4 - - [28/Feb/2015 22:14:55] "GET /favicon.ico HTTP/1.1" 404 - # ip addr show docker0: inet 172.17.42.1/16 scope docker0 [root@docker ~]# iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination ACCEPT tcp -- anywhere 172.17.0.2 tcp dpt:commplex-main [root@docker ~]# lynx http://172.17.0.2:5000 61. 61 . Inspect training/webapp Running a Web Application in Docker [root@docker01 ~]# docker top cocky_hopper UID PID PPID C STIME TTY TIME CMD root 7544 3689 0 17:12 ? 00:00:00 python app.py [root@docker01 ~]# docker inspect cocky_hopper "Cmd": [ "python", "app.py" "ExposedPorts": { "5000/tcp": {} "Hostname": "999cdee2c894", "Image": "training/webapp", "Name": "/cocky_hopper", "NetworkSettings": { "Gateway": "172.17.42.1", "IPAddress": "172.17.0.19", "IPPrefixLen": 16, "Ports": { "5000/tcp": [ "HostIp": "0.0.0.0", "HostPort": "49154" 62. 62 .Managing Data in Containers Mount a Host Directory as a Data Volume [root@docker01 ~]# docker run -d -P --name web -v /webapp training/webapp python app.py # -v, --volume=[] Bind mount a volume (e.g., from the host: -v /host:/con-tainer, from Docker: -v /container) 191388a413d843a9e6ae020b9bf051698b8755e7081e2d9eeab77a2dbb72bdd1 =========================================================================== [root@docker ~]# docker run -d -P --name web -v /src/webapp:/opt/webapp training/webapp python app.py [root@docker ~]# cd /src;ls webapp Mount a Host File as a Data Volume [root@docker]# docker run --rm -it -v ~/.bash_history:/.bash_history centos /bin/bash [root@5bf8bf23f10b /]# ls -al | more ls: cannot access .bash_history: Permission denied -?????????? ? ? ? ? ? .bash_history 63. 63 . From container Making images ( 2 methods) [root@docker01 ~]# docker run -t -i training/sinatra /bin/bash Status: Downloaded newer image for training/sinatra:latest root@62f680cfd5a4:/# gem install json ;exit Fetching: json-1.8.2.gem (100%) exit [root@docker01 ~]# docker ps -l CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 62f680cfd5a4 training/sinatra:latest "/bin/bash" 9 minutes ago Exited (0) About a minute ago angry_yonath [root@docker01 ~]# docker commit -m "Added json gem" -a "Kate Smith" 62f680cfd5a4 ouruser/sinatra:v2 52fc4cf3a3dc049ecd43f0626b53c4480305f8463461bd519c338f99a4c2743b [root@docker01 ~]# docker images REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE ouruser/sinatra v2 52fc4cf3a3dc About a minute ago 451.9 MB [root@docker01 ~]# docker run -t -i ouruser/sinatra:v2 /bin/bash root@215d1f67558b:/# 64. 64 . From Dockerfile Making images ( 2 methods) [root@docker wget]# docker build - t="oscinfra/centos:intall_wget" . Sending build context to Docker daemon 2.56 kB Sending build context to Docker daemon Step 0 : FROM oscinfra/centos:latest ---> 403871a8320a Step 1 : MAINTAINER hojin kim "khoj@osci.kr" ---> Running in 4c4bc393c67e ---> 8cc5127c853a Removing intermediate container 4c4bc393c67e Step 2 : RUN yum install -y wget ---> Running in 2ca7b10b283a Loaded plugins: fastestmirror Installed: wget.x86_64 0:1.14-10.el7_0.1 Complete! ---> 3bbded5a9761 Removing intermediate container 2ca7b10b283a Step 3 : RUN mkdir /root/wget ---> Running in 2de6060b4562 ---> 6ba1987b89a7 Removing intermediate container 2de6060b4562 Step 4 : EXPOSE 22 ---> Running in 59d051bb382d ---> c945ac8f8743 Removing intermediate container 59d051bb382d Successfully built c945ac8f874 [root@docker wget]# cat Dockerfile # for the technet seminar by hojin kim FROM oscinfra/centos:tool MAINTAINER hojin kim "khoj@osci.kr" RUN yum install -y wget RUN mkdir /root/wget EXPOSE 22 # default port [root@docker]# docker images REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE oscinfra/centos intall_wgeet c945ac8f8743 19 seconds ago 379.9 MB oscinfra/centos tool 1f06057f9152 24 minutes ago 366.5 MB oscinfra/centos latest 403871a8320a 26 minutes ago 366.5 MB [root@docker ~]# docker push oscinfra/centos:tool The push refers to a repository [oscinfra/centos] (len: 1) Sending image list Pushing repository oscinfra/centos (1 tags) 65. 65 . Uploading image file to Docker Hub Push your image to the Docker Hub Registry you@tutorial:~$ docker push learn/ping The push refers to a repository [learn/ping] (len: 1) Processing checksums Sending image list _ __ _____| | | __| | ___ _ __ ___ | | / / / _ | | / _` |/ _ | '_ / _ | | V V / __/ | | | (_| | (_) | | | | __/ |_| _/_/ ___|_|_| __,_|___/|_| |_|___| (_) ## . ## ## ## == ## ## ## ## === /""""""""""""""""___/ === ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ / ===- ~~~ ______ o __/ __/ __________/ | | __ | __ __ | _ __ _ / | / / |/ / _ | __/| __/ __ |_ __ | 66. 66 . Docker compose 구성 예 (뒤에 자세히 나옴) Docker Compose you@tutorial:~$ cat docker-compose.yml web: build: . links: - db ports: - "8000:8000" db: image: postgres you@tutorial:~$ cat Dockerfile FROM python:2.7 WORKDIR /code ADD requirements.txt /code/ RUN pip install -r requirements.txt ADD . /code CMD python app.py [root@docker01 wordpress]# docker-compose up [root@docker01 ~]# docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 7ddf0f0dec20 wordpress_web:latest "php -S 0.0.0.0:8000 4 minutes ago Up 4 minutes 0.0.0.0:8000->8000/tcp wordpress_web_1 83ddc6ea784c orchardup/mysql:latest "/usr/local/bin/run" 4 minutes ago Up 4 minutes 3306/tcp wordpress_db_1 67. 67 Contents 3. docker 구축 기초 [ 30분] 4. 오픈스택에 docker를 연결 [ 30분] 1. Openstack 인프라 구축 (4 node 구성) [ 30분] 2. Openstack 위에 VM 생성 [ 20분 ] 5. Docker로 WEB서비스 구축 [ 15분] 6. 7. Docker로 jenkins 구현 [30분] Openstack 위에 Docker로 WEB서비스 구축 [ 15분] 68. 68 What to do during 20 min. Docker + openstack Docker와 openstack의 연동 필요성 Docker와 openstack을 연동하기 출처 : https://wiki.openstack.org/wiki/Docker 69. 69 Connect to Openstack Docker prerequsite Install python-pip/git/gcc/wget/lynx Install Oslo.log yum install -y python-pip git gcc wget yum install -y docker usermod -G docker nova service openstack-nova-compute restart pip install pbr wget https://pypi.python.org/packages/source/o/oslo.log/oslo.log- 0.4.0.tar.gz#md5=e02b6feebe849c8bae50b5c329f7a9e0 tar -xvf oslo.log-0.4.0.tar.gz cd ./oslo.log-0.4.0 python setup.py install ; pip install pbr Nova-docker Install nova-docker git checkout stable/juno git clone https://github.com/stackforge/nova-docker.git cd nova-docker/ git checkout stable/juno python setup.py install 70. 70 Change openstack setting Setting env chmod 666 /var/run/docker.sock mkdir /etc/nova/rootwrap.d cat /etc/nova/rootwrap.d/docker.filters # nova-rootwrap command filters for setting up network in the docker driver # This file should be owned by (and only-writeable by) the root user [Filters] # nova/virt/docker/driver.py: 'ln', '-sf', '/var/run/netns/.*' ln: CommandFilter, /bin/ln, root service docker start chmod 660 /var/run/docker.sock mkdir /etc/nova/rootwrap.d cat /etc/nova/nova.conf # compute dirver 바꾸기 compute_driver = novadocker.virt.docker.DockerDriver cat /etc/glance/glance-api.conf # container 형식으로 지원변경 container_formats=ami,ari,aki,bare,ovf,ova,docker service openstack-glance-api restart service openstack-nova-compute restart 71. 71 Make glance image Setting env check docker Check the openstack $ docker pull busybox $ docker save busybox | glance image-create --is-public=True --container- format=docker --disk-format=raw --name busybox nova keypair-add mykey > mykey.pem nova boot --flavor m1.small --image cirros --key-name mykey test1 nova list ssh -i ../devstack/mykey.pem cirros@ docker pull busybox:latest cd source keystonerc_admin docker save busybox | glance image-create --is-public=True --container- format=docker --disk-format=raw --name busybox glance image-list nova boot --image busybox --flavor m1.tiny --nic net-id a937454d-a905- 43d2-818d-8fc5a920d8f2 busybox docker ps -a docker attach 72. 72 Check the status Setting env Docker 상태를 먼저 살펴본다. $[root@juno-compute nova-docker]# docker run -i -t fedora /bin/bash Pulling repository fedora 834629358fe2: Download complete Status: Downloaded newer image for fedora:latest bash-4.3# 간단한 이미지를 만들어본다. [root@jun https://www.friendbookmark.com/videos/977/docker-de-ponta-a-ponta-do-desenvolvimento-nuvem-unicid-novembro-2019ApresentaÃÃo sobre primeiros passos no uso de Docker, bem como dicas e truques na manipulaÃÃo de containers. Palestra realizada para profissionais da UNICID na cidade de SÃo Paulo-SP no dia 29/11/2019. Topics covered: 1. â Microsoft Most Valuable Professional (MVP) â Multi-Plataform Technical Audience Contributor (MTAC) â Mais de 15 anos de experiÃncia na Ãrea de Tecnologia â Autor TÃcnico e Palestrante â Um dos organizadores do Canal .NET, do .NET SÃo Paulo e do DevOps Professionals Renato Groffe h t t p s : / / m e d i u m . c o m / @ re n a t o . g rof f e / 2. Renato Groffe - Contatos h t t p s : / / m e d i u m . c o m / @ re n a t o . g rof f e / /renatogroffe /in/renatogroffe /canaldotnet /renatogroffe /canaldotnet /renatogroff 3. Desconto de 20%: http://bit.ly/anp-devops-fujitsu www.azurenapratica.com Black Week - Desconto de 25%: http://bit.ly/anp-serverless-unicid www.azurenapratica.com 4. Agenda â Docker: primeiros passos â Dicas e truques na utilizaÃÃo de containers Docker â Exemplos prÃticos 5. Um pouco mais sobre 6. Conceitos Fundamentais â Imagens â base para criaÃÃo de containers, sÃo geradas a partir do arquivo Dockerfile; contÃm todos os arquivos e dependÃncias de uma aplicaÃÃo â incluindo sistema operacional â Containers â pacote com tudo que à necessÃrio para a execuÃÃo de uma aplicaÃÃo (serviÃo, site, API) 7. Onde encontrar imagens? 8. Imagens Alpine â Baseadas na distribuiÃÃo Alpine Linux â PreocupaÃÃo com seguranÃa e simplicidade â Imagens Docker menores (contendo apenas o mÃnimo necessÃrio para a execuÃÃo da aplicaÃÃo) 9. Principais usos â Deployment de APIs REST e sites â Arquitetura de microserviÃos â SoluÃÃes escalÃveis atravÃs do uso de orquestradores e alternativas do tipo PaaS 10. Por que utilizar containers Docker? â Isolamento â UtilizaÃÃo mais racional de recursos â Rapidez no deployment â Menor dependÃncia do ambiente 11. E com isto temos o fim da âdesculpaâ... 12. Mas à sà isso? 13. O que mais podemos utilizar com Docker? â ServiÃos de processamento contÃnuo â Servidores de bancos de dados â Rotinas de processamento em geral 14. Algumas tecnologias com suporte a Docker 15. Montando ambientes de testes com containers â Rapidez na instalaÃÃo â DesinstalaÃÃo sem grandes dificuldades â Diferentes versÃes de um software numa mesma mÃquina 16. Criando containers Docker na prÃtica 17. Docker + SQL Server + â CriaÃÃo de containers do SQL Server 2017 e 2019 em portas diferentes 18. Criando uma imagem e publicando no Docker Hub + â Uso do NGINX para a geraÃÃo de uma imagem com um site estÃtico 19. Criando uma imagem com ASP.NET Core 3.0 â API REST criada com o ASP.NET Core 3.0 â Contagem de acessos/requisiÃÃes recebidas (publicada no Docker Hub como a imagem renatogroffe/apicontagem-3-0- alpine) 20. Um questionamento importante... â Como evitar criar separamente mÃltiplos containers, que possuem algum tipo de relaÃÃo entre si? 21. Docker Compose à a resposta! 22. Docker Compose: uma visÃo geral â CriaÃÃo e execuÃÃo conjunta dos mÃltiplos containers de uma aplicaÃÃo â Facilita o deployment em ambientes de desenvolvimento e testes â Suporte a integraÃÃo contÃnua 23. Docker Compose: uma visÃo geral â à comum a criaÃÃo de networks (redes) do Docker para deployment â Arquivo com configuraÃÃes no formato YAML (docker-compose.yml) â Suporte tambÃm do Visual Studio Code 24. Docker Compose: exemplo prÃtico CriaÃÃo de um ambiente com: â MongoDB + Mongo Express â Redis â Neo4j 25. E quanto ao suporte a Docker no Microsoft Azure? + 26. Suporte a Docker no Azure â Azure Container Instances â Azure Container Registry â Azure Web App for Containers â Azure Kubernetes Service (AKS) 27. Azure Container Instances â CriaÃÃo de containers de forma descomplicada 28. Azure Container Registry â Armazenamento de imagens Docker privadas na nuvem â Alternativa ao Docker Hub / Docker Store 29. Azure Container Registry â PublicaÃÃo docker tag apicontagem:latest groffecr.azurecr.io/apicontagem docker login groffecr.azurecr.io -u USUÃRIO -p SENHA docker push groffecr.azurecr.io/apicontagem 30. Dificuldades na adoÃÃo de containers... â Como escalar containers? â Como garantir o trabalho coordenado entre os diferentes containers de uma aplicaÃÃo? â Como detectar containers com falhas e corrigir isso automaticamente? 31. E como superar tais dificuldades? 32. Azure Web App for Containers â Hospedagem de aplicaÃÃes â Uso de Containers Docker (imagens Linux e Windows) â Suporte a Continuous Integration 33. Azure Web App for Containers â Escalabilidade (vertical e horizontal) â Suporte a HTTPS â CriaÃÃo de recursos a partir de imagens ou Docker Compose 34. Deployment Automatizado â FÃcil integraÃÃo com Azure DevOps â Build Automatizado com publicaÃÃo de imagens e deployment no Azure App Service/Azure Web App for Containers 35. Utilizando orquestradoresâ Azure Kubernetes Service (AKS) + 36. Kubernetes: uma visÃo geral â TambÃm conhecido como K8s ou kube â Desenvolvido originalmente pela Google â Mantido pela Cloud Native Computing Foundation â Escrito em Go â Open source 37. Kubernetes: uma visÃo geral â Cluster com mÃquina Master e Nodes â CriaÃÃo de objetos atravÃs de arquivos no formato YAML â Funcionalidades para gerenciamento, orquestraÃÃo e auto recuperaÃÃo de containers â kubectl â ferramenta de linha de comando â Minikube â ambiente de testes 38. Kubernetes: arquitetura kubectl 39. Kubernetes: arquitetura â Pod â Grupo de um ou mais containers implantados em um Node (NÃ) â Compartilham o mesmo endereÃo IP, IPC, nome do host e outros recursos POD 40. Kubernetes: arquitetura â Deployment â AbstraÃÃo de um Pod com recursos adicionais â Conta com gerenciamento de estados Deployment POD 41. Kubernetes: arquitetura â Service â Objeto mais estÃvel (Pods sÃo criados ou removidos continuamente) â Cuidarà do acesso aos Pods, funcionando como um Load Balancer 42. Kubernetes: arquitetura 43. Exemplo PrÃtico â API REST criada com o ASP.NET Core 3.0 â Serà utilizada a mesma API de contagem de acessos dos exemplos anteriores (publicada no Docker Hub como a imagem renatogroffe/apicontagem-3-0-alpine) â CriaÃÃo de um cluster atravÃs do Azure Kubernetes Service (AKS) 44. medium.com/@renato.groffe/ https://www.friendbookmark.com/videos/976/microservices-docker-kubernetes-istioIntroduction to Microservices Architecture, Docker, Kubernetes, Istio, Testing Strategies for Microservices based Apps. Security Best Practices. Kanban, DevOps, and SRE. Infrastructure Design Patterns - API Gateway - Service Discovery - Load Balancer - Circuit Breaker - Let-it-Crash Pattern Software Design Patterns - Hexagonal Architecture - Domain Driven Design - Event Sourcing and CQRS - Functional Reactive Programming Topics covered: 1. Microservices Architecture Building Cloud Native Apps Design Patterns, Containers, Kubernetes, Istio, Kafka, Saga â Distributed Transactions, Testing, Security, Kanban SRE, DevOps ARAF KARSH HAMID Co-Founder / CTO MetaMagic Global Inc., NJ, USA @arafkarsh arafkarsh 2. 2Slides are color coded based on the topic colors. Microservices Containers & Kubernetes, Kafka 1 Infrastructure Patterns Capability Centric Design DDD / ES & CQRS Reactive Programming 2 Testing Strategies Security Best Practices 3 Agile: Kanban ITSM, DevOps, SRE 4 3. MICROSERVICES â CONCEPTS â WHEN SHOULD YOU USE MICROSERVICES? â WHATâS THE RIGHT SIZE? â ARCHITECTURE (INFRATRUCTURE AND SOFTWRE) 4/1/2019 3 1 4. 4Source: https://cloud.google.com/kubernetes-engine/kubernetes-comic/ 5. Pioneers in Microservices Implementation 01-04-2019 5 New Entrants 6. 6 100s Microservices 1,000s Releases / Day 10,000s Virtual Machines 100K+ User actions / Second 81 M Customers Globally 1 B Time series Metrics 10 B Hours of video streaming every quarter Source: NetFlix: : https://www.youtube.com/watch?v=UTKIT6STSVM 10s OPs Engineers 0 NOC 0 Data Centers So what do NetFlix think about DevOps? No DevOps Donât do lot of Process / Procedures Freedom for Developers & be Accountable Trust people you Hire No Controls / Silos / Walls / Fences Ownership â You Build it, You Run it. 7. 7 50M Paid Subscribers 100M Active Users 60 Countries Cross Functional Team Full, End to End ownership of features Autonomous1000+ Microservices Source: https://microcph.dk/media/1024/conference-microcph-2017.pdf 1000+ Tech Employees 120+ Teams 8. Microservices definition 4/1/2019 8 In short, the microservice architectural style is an approach to developing a single application as a suite of small services, each running in its own process and communicating with lightweight mechanisms, often an HTTP resource API. These services are built around business capabilities and independently deployable by fully automated deployment machinery. There is a bare minimum of centralized management of these services, which may be written in different programming languages and use different data storage technologies. https://martinfowler.com/articles/microservices.html By James Lewis and Martin Fowler Bolo Definition Kya hai? Tell me whatâs the definition 9. Microservices Characteristics 9 We can scale our operation independently, maintain unparalleled system availability, and introduce new services quickly without the need for massive reconfiguration. â Werner Vogels, CTO, Amazon Web Services Modularity ... is to a technological economy what the division of labor is to a manufacturing one. W. Brian Arthur, author of e Nature of Technology The key in making great and growable systems is much more to design how its modules communicate rather than what their internal properties and behaviors should be. Alan Kay, 1998 email to the Squeak-dev list Components via Services Organized around Business Capabilities Products NOT Projects Smart Endpoints & Dumb Pipes Decentralized Governance & Data Management Infrastructure Automation Design for Failure Evolutionary Design 10. When should I use them (Microservices)? 01-04-2019 10 â Strong Module Boundaries: Microservices reinforce modular structure, which is particularly important for larger teams. â Independent Deployment: Simple services are easier to deploy, and since they are autonomous, are less likely to cause system failures when they go wrong. â Technology Diversity: With microservices you can mix multiple languages, development frameworks and data- storage technologies. When you have Whatâs the Cost Distribution: Distributed systems are harder to program, since remote calls are slow and are always at risk of failure. Eventual Consistency: Maintaining strong consistency is extremely difficult for a distributed system, which means everyone has to manage eventual consistency. Operational Complexity: You need a mature operations team to manage lots of services, which are being redeployed regularly. Source: https://www.martinfowler.com/microservices/ 11. What is the right size for a Microservice? 01-04-2019 11 â Rather than the size what matters is the Business Function / Domain of the service. â One Microservice may have half a dozen entities and other a couple of dozen entities. Whatâs more important is the role Microservices plays. â Bounded Context from DDD helps you to decompose a large multi domain Monolith into a Microservice for each Bounded Context. â Focusing on User stories will help you clearly define the boundaries of the Business Domain. 12. Microservices Architecture / Design Patterns 01-04-2019 12 â API Gateway â Service Discovery â Load Balancer â Config Service â Circuit Breaker â Service Mesh â Event Bus / Streams â Hexagonal Architecture â Domain Driven Design â Event Sourcing & CQRS â Functional Reactive Programming â MVC, MV*, Redux Infrastructure Architecture 13. 13 Monolithic vs. Microservices Example Traditional Monolithic App using Single Technology Stack Micro Services with Multiple Technology Stack This 3 tier model is obsolete now. Source: Gartner Market Guide for Application Platforms. Nov 23, 2016 Event Stream / Queues / Pub-Sub / Storage UI Layer Web Services Business Logic Database Layer Micro Service 4 EE 7 Inventory UI Layer Web Services Business Logic Database Layer Micro Service 1 Customer SE 8 UI Layer Web Services Business Logic Database Layer Micro Service 3 ShoppingCart UI Layer Web Services Business Logic Database Layer Micro Service 2 Order UI Layer WS BL DL Database ShoppingCart Order Customer Inventory API Gateway (Zuul Edge Server) Load Balancer (Ribbon) Circuit Breaker (Hystrix) Service Discovery (Eureka) Load Balancer (Ribbon) Circuit Breaker (Hystrix) Load Balancer (Ribbon) Circuit Breaker (Hystrix) Load Balancer (Ribbon) Circuit Breaker (Hystrix) 12 14. 01-04-2019 14 SOA vs. Microservices Example Traditional Monolithic App with SOA Micro Services with Multiple Technology Stack Event Stream / Queues/ Pub-Sub / Storage UI Layer Web Services Business Logic Database Layer Micro Service 1 Customer SE 8 UI Layer Web Services Business Logic Database Layer Micro Service 3 ShoppingCart UI Layer Web Services Business Logic Database Layer Micro Service 2 Order API Gateway Load Balancer Circuit Breaker Service Discovery Load Balancer Circuit Breaker Load Balancer Circuit Breaker UI Layer Database ShoppingCart Order Customer Inventory Enterprise Service Bus Messaging REST / SOAP HTTP MOM JMS ODBC / JDBC Translation Web Services XML WSDL Addressing Security Registry Management Producers Shared Database Consumers3rd Party Apps Smart Pipes Lot of Business logic resides in the Pipe 15. Microservices Deployment Model Microservices with Multiple Technology Stack â Software Stack for Networking Event Stream / Queues / Pub-Sub / Storage Users Service Discovery (Eureka) Config Server (Spring) API (Zuul) Gateway UI Layer Web Services Business Logic Database Layer Micro Service 2 ShoppingCart SE 8 LB = Ribbon CB = Hystrix LB = Ribbon CB = Hystrix UI Layer Web Services Business Logic Database Layer Product SE 8 Micro Service 1 With 4 node cluster LB = Ribbon CB = Hystrix UI Layer Web Services Business Logic Database Layer Order SE 8 Micro Service 3 With 2 node Cluster LB = Ribbon CB = Hystrix UI Layer Web Services Business Logic Database Layer Customer Micro Service 4 With 2 node cluster HTTP Server All UI Code is bundled Virtual Private Network 01-04-2019 15 16. Shopping Portal â Docker / Kubernetes â Network Stack /ui /productms Load Balancer Ingress Deployment / Replica / Pod NodesKubernetes Objects Firewall UI Pod UI Pod UI Pod UI Service N1 N2 N2 EndPoints Product Pod Product Pod Product Pod Product Service N4 N3 MySQL Pod EndPoints Review Pod Review Pod Review Pod Review Service N4 N3 N1 Service Call Kube DNS EndPoints Internal Load Balancers 16 Users Routing based on Layer 3,4 and 7 17. 17 Source:https://cloud.google.com/kubernetes-engine/kubernetes-comic/ Each one of the Microservices can now be â Debugged, â Updated, and â Deployed individually without the whole Project coming to a standstill. An important step on the path to â Continuous Integration and â Continuous Delivery. 18. 18 Source:https://cloud.google.com/kubernetes-engine/kubernetes-comic/ 19. 19Source: https://cloud.google.com/kubernetes-engine/kubernetes-comic/ 20. 20 12FactorAppMethodology 4 Backing Services Treat Backing services like DB, Cache as attached resources 5 Build, Release, Run Separate Build and Run Stages 6 Process Execute App as One or more Stateless Process 7 Port Binding Export Services with Specific Port Binding 8 Concurrency Scale out via the process Model 9 Disposability Maximize robustness with fast startup and graceful exit 10 Dev / Prod Parity Keep Development, Staging and Production as similar as possible Checkout the Shift â Left in DevOps (Slide 157) 11 Logs Treat logs as Event Streams 12 Admin Process Run Admin Tasks as one of Process (Eg. DB Migration, Software upgrades, etc..) Factors Description 1 Codebase One Code base tracked in revision control 2 Dependencies Explicitly declare dependencies 3 Configuration Configuration driven Apps Source:https://12factor.net/ 21. Catalogues of Microservices 4/1/2019 21 System Z Model From Spotify â Different types of Components Z Supports â Libraries â Data Pipelines â Views in the client â Data Store â Service 22. Pros 1. Adds Complexity 2. Skillset shortage 3. Confusion on getting the right size 4. Team need to manage end-to-end of the Service (From UI to Backend to Running in Production). 01-04-2019 22 1. Robust 2. Scalable 3. Testable (Local) 4. Easy to Change and Replace 5. Easy to Deploy 6. Technology Agnostic Cons Microservices Pros and Cons 23. Monolithic > Microservices â FORRESTER RESEARCH â MODERNIZATION JOURNEY â ASSESS AND CLASSIFY YOUR APP PORTFOLIO â PLAN AND PRIORITIZE 4/1/2019 23 24. Conwayâs Law 4/1/2019 (C)COPYRIGHTMETAMAGICGLOBALINC.,NEWJERSEY,USA 24 Any Organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organizationâs communication structure. 25. * For IT Services : They can do one more project of same size with ZERO COST in Platform Licensing (Based on 20 developer pack USD $50K per month for 3 months License cost = $150K) 2501-04-2019 26. Scale Cube and Micro Services 4/1/2019 26 1. Y Axis Scaling â Functional Decomposition : Business Function as a Service 2. Z Axis Scaling â Database Partitioning : Avoid locks by Database Sharding 3. X Axis Scaling â Cloning of Individual Services for Specific Service Scalability 27. Modernization Journey 4/1/2019 27  Start new features as Microservices Incrementally establish the success early.  Expose Legacy On-Premise Apps APIâs If legacy Apps cant be shifted to Cloud  Refactor Monolithic features to Microservices Breakdown and Deploy Feature by Feature  Containerize the Microservice Reduce costs, simplifies the operations and consistent environment between Dev, QA and Production  Monolith De-commission Plan Incrementally sunset the Monolith Velocity as you transform Increase your Delivery Velocity along the Journey High FuturePresent Low Inspired by a paper from IBM 28. Assess and Classify your App Portfolio 4/1/2019 28  Take inventory of your Apps Classify the Apps based on technology, complexity.  Align Apps to your Business Priorities Identify the Apps that are critical for Modernization.  Identify Business Modernization Requirements Create a Roadmap with faster go to market strategy  Understand the effort and Cost Evaluate all possible Modernization options Container Refactor Expose APIsLift & Shift BusinessValueCostComplexity Product Catalogue Product Review Inventory Shopping Cart Customer Profile Order Management Inspired by a paper from IBM 29. Plan and Prioritize 4/1/2019 29 Complexity Cost Value Score Rank Weightage 35% 25% 40% Customer Med 3 Med 3 Low 1 2.20 7 6 Product Reviews Med 3 High 5 Med 3 3.50 11 3 Product Catalogue Med 3 Med 3 High 5 4.80 11 1 Shopping Cart High 5 Med 3 Med 3 3.70 11 4 Order Very High 7 Med 3 High 5 5.20 15 2 Inventory Very High 7 High 5 Med 3 4.90 15 5  Prioritize Low Priority projects are good test cases but does not bring much business value.  Quick Wins Identify a feature / project which has good business value and low in complexity.  Project Duration Focus on shorter duration projects with high Business Value. Shopping Portal Features Prioritization Inspired by a paper from IBM 30. Monolithic to Microservices Summary 4/1/2019 30 1. Classify your Apps into Following areas 1. Lift and Shit 2. Containerize 3. Refactor 4. Expose API 2. Prioritize High Business Value Low Technical Complexity 3. Focus on Shorter Duration â From Specs to Operation 31. Containers & Kubernetes â DOCKER CONTAINERS â KUBERNETES â CONTAINER ORCHESTRATION â ISTIO â TRAFFIC MANAGEMENT / NETWORK POLICIES 4/1/2019 (C)COPYRIGHTMETAMAGICGLOBALINC.,NEWJERSEY,USA 31 32. Servers / Virtual Machines / Containers Hardware OS BINS / LIB App 1 App 2 App 3 Server Hardware Host OS HYPERVISOR App 1 App 2 App 3 Guest OS BINS / LIB Guest OS BINS / LIB Guest OS BINS / LIB Type 1 Hypervisor Hardware Host OS App 1 App 2 App 3 BINS / LIB BINS / LIB BINS / LIB Container Hardware HYPERVISOR App 1 App 2 App 3 Guest OS BINS / LIB Guest OS BINS / LIB Guest OS BINS / LIB Type 2 Hypervisor01-04-2019 32 33. Docker containers are Linux Containers CGROUPS NAME SPACES Copy on Write DOCKER CONTAINER â Kernel Feature â Groups Processes â Control Resource Allocation â CPU, CPU Sets â Memory â Disk â Block I/O â Images â Not a File System â Not a VHD â Basically a tar file â Has a Hierarchy â Arbitrary Depth â Fits into Docker Registry â The real magic behind containers â It creates barriers between processes â Different Namespaces â PID Namespace â Net Namespace â IPC Namespace â MNT Namespace â Linux Kernel Namespace introduced between kernel 2.6.15 â 2.6.26 docker runlxc-start 33 https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/resource_management_guide/ch01 01-04-2019 34. Linux Kernel 34 HOST OS (Ubuntu) Client Docker Daemon Cent OS Alpine Debian LinuxKernel Host Kernel Host Kernel Host Kernel All the containers will have the same Host OS Kernel If you require a specific Kernel version then Host Kernel needs to be updated HOST OS (Windows 10) Client Docker Daemon Nano Server Server Core Nano Server WindowsKernel Host Kernel Host Kernel Host Kernel Windows Kernel 35. Docker DaemonDocker Client How Docker worksâ. $ docker search â. $ docker build â. $ docker container create .. Docker Hub Images Containers $ docker container run .. $ docker container start .. $ docker container stop .. $ docker container ls .. $ docker push â. $ docker swarm .. 01-04-2019 35 21 34 1. Search for the Container 2. Docker Daemon Sends the request to Hub 3. Downloads the image 4. Run the Container from the image 36. Kubernetes Key Concepts 4/1/2019 (C)COPYRIGHTMETAMAGICGLOBALINC.,NEWJERSEY,USA 36 o Declarative Model Infrastructure as a code o Desired State Required state of your App / Microservice o Current State Current State of the App due to error or load factor Pods Replicas Deploy Service 37. Deployment â Updates and rollbacks, Canary Release D ReplicaSet â Self Healing, Scalability, Desired State R Worker Node 1 Master Node (Control Plane) Kubernetes Architecture POD POD itself is a Linux Container, Docker container will run inside the POD. PODs with single or multiple containers (Sidecar Pattern) will share Cgroup, Volumes, Namespaces of the POD. (Cgroup / Namespaces) Scheduler Controller Manager Using yaml or json declare the desired state of the app. State is stored in the Cluster store. Self healing is done by Kubernetes using watch loops if the desired state is changed. POD POD POD BE 1.210.1.2.34 BE 1.210.1.2.35 BE 1.210.1.2.36 BE 15.1.2.100 DNS: a.b.com 1.2 Service Pod IP Address is dynamic, communication should be based on Service which will have routable IP and DNS Name. Labels (BE, 1.2) play a critical role in ReplicaSet, Deployment, & Services etc. Cluster Store etcd Key Value Store Pod Pod Pod Label Selector selects pods based on the Labels. Label Selector Label Selector Label Selector Node Controller End Point Controller Deployment Controller Pod Controller â. Labels Internet Firewall K8s Cluster Cloud Controller For the cloud providers to manage nodes, services, routes, volumes etc. Kubelet Node Manager Container Runtime Interface Port 10255 gRPC ProtoBuf Kube-Proxy Network Proxy TCP / UDP Forwarding IPTABLES / IPVS Allows multiple implementation of containers from v1.7 RESTful yaml / json $ kubectl â. Port 443API Server Pod IP ...34 ...35 ...36EP â Declarative Model â Desired State Key Aspects Namespace1Namespace2 â Pods â ReplicaSet â Deployment â Service â Endpoints â StatefulSet â Namespace â Resource Quota â Limit Range â Persistent Volume Kind Secrets Kind â apiVersion: â kind: â metadata: â spec: Declarative Model â Pod â ReplicaSet â Service â Deployment â Virtual Service â Gateway, SE, DR â Policy, MeshPolicy â RbaConfig â Prometheus, Rule, â ListChekcer â @ @ Annotations Names Cluster IP Node Port Load Balancer External Name @ Ingress 01-04-2019 37 38. Service Mesh â Sidecar Design Pattern 01-04-2019 38 CB â Circuit Breaker LB â Load Balancer SD â Service Discovery Microservice Process1Process2 Service Mesh Control Plane Service Discovery Routing Rules Control Plane will have all the rules for Routing and Service Discovery. Local Service Mesh will download the rules from the Control pane will have a local copy. Service Discovery Calls Service Mesh Calls Customer Microservice Application Localhost calls http://localhost/api/order/ Router Network Stack LBCB SD ServiceMesh Sidecar UI Layer Web Services Business Logic Order Microservice Application Localhost calls http://localhost/api/payment/ Router Network Stack LBCB SD ServiceMesh Sidecar UI Layer Web Services Business Logic Data Plane 39. Shopping Portal /ui /productms /auth /order Gateway Virtual Service Deployment / Replica / Pod NodesIstio Sidecar - Envoy Load Balancer Kubernetes Objects Istio Objects Firewall P M CIstio Control Plane UI Pod N5v2Canary v2User X = Canary Others = Stable A / B Testing using Canary Deployment v1 UI Pod UI Pod UI Pod UI Service N1 N2 N2 Destination Rule Stable / v1 EndPoints Internal Load Balancers 39 Source:https://github.com/meta-magic/kubernetes_workshop Users Product Pod Product Pod Product Pod Product Service MySQL Pod N4 N3 Destination Rule EndPoints Review Pod Review Pod Review Pod Review Service N1 N4 N3 Service Call Kube DNS EndPoints 40. Shopping Portal /ui /productms /auth /order Gateway Virtual Service Deployment / Replica / Pod NodesIstio Sidecar - Envoy Load Balancer Kubernetes Objects Istio Objects Firewall P M CIstio Control Plane UI Pod N5v2Canary v2 v1 UI Pod UI Pod UI Pod UI Service N1 N2 N2 Destination Rule Stable / v1 EndPoints Internal Load Balancers 40 Source:https://github.com/meta-magic/kubernetes_workshop Users Product Pod Product Pod Product Pod Product Service MySQL Pod N4 N3 Destination Rule EndPoints Review Pod Review Pod Review Pod Review Service N1 N4 N3 Service Call Kube DNS EndPoints Traffic Shifting Canary Deployment 10% = Canary 90% = Stable 41. Shopping Portal /ui /productms /auth /order Gateway Virtual Service Deployment / Replica / Pod NodesIstio Sidecar - Envoy Load Balancer Kubernetes Objects Istio Objects Firewall P M CIstio Control Plane UI Pod N5 v2Canary v2 v1 UI Pod UI Pod UI Pod UI Service N1 N2 N2 Destination Rule Stable / v1 EndPoints Internal Load Balancers 41 Source:https://github.com/meta-magic/kubernetes_workshop Users Product Pod Product Pod Product Pod Product Service MySQL Pod N4 N3 Destination Rule EndPoints Review Pod Review Pod Review Pod Review Service N1 N4 N3 Service Call Kube DNS EndPoints Blue Green Deployment 100% = Stable When you want to shift to v2 Change 100% to Canary 42. Shopping Portal /ui /productms /auth /order Gateway Virtual Service Deployment / Replica / Pod NodesIstio Sidecar - Envoy Load Balancer Kubernetes Objects Istio Objects Firewall P M CIstio Control Plane UI Pod N5 v2Canary v2 v1 UI Pod UI Pod UI Pod UI Service N1 N2 N2 Destination Rule Stable / v1 EndPoints Internal Load Balancers 42 Source:https://github.com/meta-magic/kubernetes_workshop Users Product Pod Product Pod Product Pod Product Service MySQL Pod N4 N3 Destination Rule EndPoints Review Pod Review Pod Review Pod Review Service N1 N4 N3 Service Call Kube DNS EndPoints Mirror Deployment 100% = Stable Mirror = Canary Production Data is mirrored to new release for real-time testing 43. 43 Shopping Portal /ui /productms /auth /order Gateway Virtual Service Deployment / Replica / Pod NodesIstio Sidecar - Envoy Load Balancer Firewall P M CIstio Control PlaneFault Injection MySQL Pod N4 N3 Destination Rule Product Pod Product Pod Product Pod Product Service Service Call Kube DNS EndPoints Internal Load Balancers 43 Source:https://github.com/meta-magic/kubernetes_workshop Fault Injection Delay = 2 Sec Abort = 10% Kubernetes Objects Istio Objects Users Review Pod Review Pod Review Pod Review Service N1 N4 N3EndPoints UI Pod UI Pod UI Pod UI Service N1 N2 N2 Destination Rule v1EndPoints 44. Container & Kubernetes Summary 4/1/2019 44 1. Containers are NOT Virtual Machines 2. Containers are isolated area in the OS kernel 3. Kubernetes â is a Container Orchestration Platform. 4. Kubernetes abstracts the cloud vendor (AWS, Azure, GCP) scalability features. 5. Kubernetes Concepts â Declarative Model, Desired State and Current State. 45. Kafka â CONCEPTS : QUEUES / PUB â SUB / EVENT STREAMING â WHY IS IT DIFFERENT FROM TRADITIONAL MESSAGE QUEUES? â DATA STORAGE / CLUSTER / DURABILITY â PERFORMANCE 4/1/2019 (C)COPYRIGHTMETAMAGICGLOBALINC.,NEWJERSEY,USA 45 46. Kafka Core Concepts 01-04-2019 46 Publish & Subscribe Read and write streams of data like a messaging system Process Write scalable stream processing apps that react to events in real- time. Store Store streams of data safely in a distributed, replicated, fault tolerant cluster. 47. Traditional Queue / Pub-Sub Vs. Kafka 01-04-2019 47 0 1 2 3 4 5 6 7 8 9 8 7 9 Consumer 1 Consumer 2 Consumer 3 Queues Data Data can be partitioned for scalability for parallel processing by same type of consumers Pros: Cons: Queues are NOT multi subscribers. Once a Consumer reads the data, its gone from the queue. Ordering of records will be lost in asynchronous parallel processing. 0 1 2 3 4 5 6 7 8 9 9 9 9 Consumer 1 Consumer 2 Consumer 3 Pub â Sub Data Multiple subscribers can get the same data.Pros: Scaling is difficult as every message goes to every subscriber. Cons: Kafka generalizes these two concepts.  As with a queue the consumer group allows you to divide up processing over a collection of processes (the members of the consumer group).  As with publish-subscribe, Kafka allows you to broadcast messages to multiple consumer groups. 48. Anatomy of a Topic 01-04-2019 48 Source : https://kafka.apache.org/intro â A Topic is a category or feed name to which records are published. â Topics in Kafka are always multi subscriber. â That is, a Topic can have zero, one, or many consumers that subscribe to the data written to it. â Each Partition is an ordered, immutable sequence of records that is continually appended toâa structured commit log. â A Partition is nothing but a directory of Log Files â The records in the partitions are each assigned a sequential id number called the offset that uniquely identifies each record within the partition. 49. 01-04-2019 49 Partition Log Segment â Partition (Kafkaâs Storage unit) is Directory of Log Files. â A partition cannot be split across multiple brokers or even multiple disks â Partitions are split into Segments â Segments are two files: 000.log & 000.index â Segments are named by their base offset. The base offset of a segment is an offset greater than offsets in previous segments and less than or equal to offsets in that segment. â Indexes store offsets relative to its segments base offset â Indexes map each offset to their message position in the log and they are used to look up messages. â Purging of data is based on oldest segment and one segment at a time. 0 1 2 3 4 5 6 7 8 9 Partition Data 6 3 0 Segment 0 Segment 3 Segment 6 9 Segment 9 - Active $ tree kafka-logs | head -n 6 kafka-logs |──── SigmaHawk-2 | |──── 00000000006109871597.index | |──── 00000000006109871597.log | |──── 00000000007306321253.index | |──── 00000000007306321253.log Topic / Partition Segment 1 Segment 2 Rel.Offset, Position Offset, Position, Size, Payload 0000.index 0000.log 0 0 0 0 7 ABCDEFG 1 7 1 7 4 ABCD 2 11 2 11 9 ABCDEFGIJ 4 Bytes 4 Bytes 50. 01-04-2019 50 Kafka Cluster â Topics & Partitions â The partitions of the log are distributed over the servers in the Kafka cluster with each server handling data and requests for a share of the partitions. Source : https://kafka.apache.org/intro Broker 1 Leader Broker 2 Follower Broker 3 Follower Broker 4 Follower Broker 5 Leader Partition 1 Partition 0 Topic ABC â Each server acts as a leader for some of its partitions and a follower for others so load is well balanced within the cluster. â Each partition has one server which acts as the "leader" and zero or more servers which act as "followers". 51. 01-04-2019 51 Record Commit Process Broker 1 Leader Topic 1 Broker 2 Follower Producer Consumer 2 2 Commit 3 ack â Each partition is replicated across a configurable number of servers for fault tolerance. â The leader handles all read and write requests for the partition while the followers passively replicate the leader. â If the leader fails, one of the followers will automatically become the new leader.1 Message with Offset 4 777743 Broker 3 Follower Data Durability From Kafka v0.8.0 onwards acks Acknowledgement Description 0 If set to zero then the producer will NOT wait for any acknowledgment from the server at all. The record will be immediately added to the socket buffer and considered sent. No guarantee can be made that the server has received the record in this case, and the retries configuration will not take effect (as the client won't generally know of any failures). The offset given back for each record will always be set to -1. 1 This will mean the leader will write the record to its local log but will respond without awaiting full acknowledgement from all followers. In this case should the leader fail immediately after acknowledging the record but before the followers have replicated it then the record will be lost. All / -1 This means the leader will wait for the full set of in-sync replicas to acknowledge the record. This guarantees that the record will not be lost as long as at least one in-sync replica remains alive. This is the strongest available guarantee. This is equivalent to the acks=-1 setting. Source: https://kafka.apache.org/documentation/#topicconfigs acks Steps 0 1 1 1,3 -1 1,2,3 Producer Configuration 52. 01-04-2019 52 Replication 6 3.2 m1 m2 m3 L(A) m1 m2 F(B) m1 F(C)ISR = (A, B, C) Leader A commits Message m1. Message m2 & m3 not yet committed. 1 m1 m2 F(C) m1 m2 L(B) m1 m2 m3 L(A) ISR = (B,C) A fails and B is the new Leader. B commits m22 m1 m2 m3 L(A) m1 m2 L(B) m4 m5 m1 m2 F(C) m4 m5 ISR = (B,C) B commits new messages m4 and m5 3 m1 m2 L(B) m4 m5 m1 m2 F(C) m4 m5 m1 F(A) ISR = (A, B,C) A comes back, restores to last commit and catches up to latest messages. 4 m1 m2 L(B) m4 m5 m1 m2 F(C) m4 m5 m1 m2 F(A) m4 m5 ISR â In-sync Replica â Instead of majority vote, Kafka dynamically maintains a set of in-sync replicas (ISR) that are caught-up to the leader. â Only members of this set are eligible for election as leader. â A write to a Kafka partition is not considered committed until all in-sync replicas have received the write. â This ISR set is persisted to ZooKeeper whenever it changes. Because of this, any replica in the ISR is eligible to be elected leader. 53. LinkedIn Kafka Cluster 01-04-2019 53 Brokers60 Partitions50K Messages / Second800K MB / Second inbound300 MB / Second Outbound1024 The tuning looks fairly aggressive, but all of the brokers in that cluster have a 90% GC pause time of about 21ms, and theyâre doing less than 1 young GC per second. 54. Uber Kafka Cluster 01-04-2019 54 Topics10K+ Events / Second11M Petabytes of Data1PB+ 55. Kafka Summary 4/1/2019 55 1. Combined Best of Queues and Pub / Sub Model. 2. Data Durability 3. Fastest Messaging Infrastructure 4. Streaming capabilities 5. Replication 56. 4/1/2019 56 Architecture & Design Patterns â I N F R A ST R U C T U R E D ES I G N PAT T E R N S â C A PA B I L I T Y C E N T R I C D ES I G N â D O M A I N D R I V E N D ES I G N â E V E N T S O U RC I N G & CQ RS â F U N C T I O NA L R EAC T I V E P RO G R A M M I N G â U I D ES I G N PAT T E R N S â R EST F U L A P I S A N D V E RS I O N I N G 2 57. 4/1/2019 57 Infrastructure Design Patterns â API GATEWAY â LOAD BALANCER â SERVICE DISCOVERY â CIRCUIT BREAKER â SERVICE AGGREGATOR â LET-IT CRASH PATTERN 58. API Gateway Design Pattern 58 UILayer WS BL DL Database Shopping Cart Order Customer Product Firewall Users API Gateway LoadBalancer CircuitBreaker UILayer WebServices BusinessLogic DatabaseLayer Product SE8 Product Microservice With 4 node cluster LoadBalancer CircuitBreaker UILayer WebServices BusinessLogic DatabaseLayer Customer Customer Microservice With 2 node cluster Users Access the Monolithic App Directly API Gateway (Reverse Proxy Server) routes the traffic to appropriate Microservices (Load Balancers) 59. Load Balancer Design Pattern 59 Firewall Users API Gateway Load Balancer CircuitBreaker UILayer WebServices BusinessLogic DatabaseLayer Product SE8 Product Microservice With 4 node cluster Load Balancer CB=Hystrix UILayer WebServices BusinessLogic DatabaseLayer Customer Customer Microservice With 2 node cluster API Gateway (Reverse Proxy Server) routes the traffic to appropriate Microservices (Load Balancers) Load Balancer Rules 1. Round Robin 2. Based on Availability 3. Based on Response Time 60. Service Discovery â NetFlix Network Stack Model 60 Firewall Users API Gateway LoadBalancer CircuitBreaker Product Product Microservice With 4 node cluster LoadBalancer CircuitBreaker UILayer WebServices BusinessLogic DatabaseLayer Customer Customer Microservice With 2 node cluster â In this model Developers write the code in every Microservice to register with NetFlix Eureka Service Discovery Server. â Load Balancers and API Gateway also registers with Service Discovery. â Service Discovery will inform the Load Balancers about the instance details (IP Addresses). Service Discovery 61. Service Discovery â Kubernetes Model Kubernetes Objects Firewall Service Call Kube DNS 61 Users Sports 1 Sports 2 Sports 3 Sports Service N4 N3 N1 EndPoints Internal Load Balancers DB Reverse Proxy Server API Gateway N1 N2 N2Politics 1 Politics 2 Politics 3 Politics Service EndPoints DB Internal Load Balancers Pods Nodes â API Gateway (Reverse Proxy Server) doesn't know the instances (IP Addresses) of News Pod. It knows the IP address of the Services defined for each Microservice (News, Politics, Sports etc.). â Services handles the dynamic IP Addresses of the pods. Services will automatically discover the new Pods based on Labels. Service Definition from Kubernetes Perspective Internal Load Balancers EndPoints News Pod 1 News Pod 2 News Pod 3 News Service N4 N3 N2 Pods Nodes 62. Circuit Breaker Pattern /ui /productms If Product Review is not available Product service will return the product details with a message review not available. Reverse Proxy Server Ingress Deployment / Replica / Pod NodesKubernetes Objects Firewall UI Pod UI Pod UI Pod UI Service N1 N2 N2 EndPoints Product Pod Product Pod Product Pod Product Service N4 N3 MySQL Pod EndPoints Internal Load Balancers 62 Users Routing based on Layer 3,4 and 7 Review Pod Review Pod Review Pod Review Service N4 N3 N1 Service Call Kube DNS EndPoints 63. Service Aggregator Pattern /newservice Reverse Proxy Server Ingress Deployment / Replica / Pod Nodes Kubernetes Objects Firewall Service Call Kube DNS 63 Users Internal Load Balancers EndPoints News Pod News Pod News Pod News Service N4 N3 N2 News Service Portal â News Category wise Microservices â Aggregator Microservice to aggregate all category of news. Auto Scaling â Sports Events (IPL) spikes the traffic for Sports Microservice. â Auto scaling happens for both News and Sports Microservices. N1 N2 N2National National National National Service EndPoints Internal Load Balancers DB N1 N2 N2Politics Politics Politics Politics Service EndPoints DB Sports Sports Sports Sports Service N4 N3 N1 EndPoints Internal Load Balancers DB 64. Music UI 4/1/2019 64 Play Count Discography Albums 65. Service Aggregator Pattern /artist Reverse Proxy Server Ingress Deployment / Replica / Pod Nodes Kubernetes Objects Firewall Service Call Kube DNS 65 Users Internal Load Balancers EndPoints Artist Pod Artist Pod Artist Pod Artist Service N4 N3 N2 Spotify Microservices â Artist Microservice combines all the details from Discography, Play count and Playlists. Auto Scaling â Scaling of Artist and downstream Microservices will automatically scale depends on the load factor. N1 N2 N2Discography Discography Discography Discography Service EndPoints Internal Load Balancers DB N1 N2 N2Play Count Play Count Play Count Play Count Service EndPoints DB Playlist Playlist Playlist Playlist Service N4 N3 N1 EndPoints Internal Load Balancers DB 66. Software Network Stack Vs Network Stack 4/1/2019 66 Pattern Software Stack Java Software Stack .NET Kubernetes 1 API Gateway Zuul Server SteelToe Istio Envoy 2 Service Discovery Eureka Server SteelToe Kube DNS 3 Load Balancer Ribbon Server SteelToe Istio Envoy 4 Circuit Breaker Hysterix SteelToe 5 Config Server Spring Config SteelToe Secrets, Env - K8s Master Web Site https://netflix.github.io/ https://steeltoe.io/ https://kubernetes.io/ Developer need to write code to integrate with the Software Stack (Programming Language Specific. For Ex. Every microservice needs to subscribe to Service Discovery when the Microservice boots up. Service Discovery in Kubernetes is based on the Labels assigned to Pod and Services and its Endpoints (IP Address) are dynamically mapped (DNS) based on the Label. 67. Let-it-Crash Design Pattern â Erlang Philosophy 4/1/2019 67 â The Erlang view of the world is that everything is a process and that processes can interact only by exchanging messages. â A typical Erlang program might have hundreds, thousands, or even millions of processes. â Letting processes crash is central to Erlang. Itâs the equivalent of unplugging your router and plugging it back in â as long as you can get back to a known state, this turns out to be a very good strategy. â To make that happen, you build supervision trees. â A supervisor will decide how to deal with a crashed process. It will restart the process, or possibly kill some other processes, or crash and let someone else deal with it. â Two models of concurrency: Shared State Concurrency, & Message Passing Concurrency. The programming world went one way (toward shared state). The Erlang community went the other way. â All languages such as C, Java, C++, and so on, have the notion that there is this stuff called state and that we can change it. The moment you share something you need to bring Mutex a Locking Mechanism. â Erlang has no mutable data structures (thatâs not quite true, but itâs true enough). No mutable data structures = No locks. No mutable data structures = Easy to parallelize. 68. Let-it-Crash Design Pattern 4/1/2019 68 1. The idea of Messages as the first class citizens of a system, has been rediscovered by the Event Sourcing / CQRS community, along with a strong focus on domain models. 2. Event Sourced Aggregates are a way to Model the Processes and NOT things. 3. Each component MUST tolerate a crash and restart at any point in time. 4. All interaction between the components must tolerate that peers can crash. This mean ubiquitous use of timeouts and Circuit Breaker. 5. Each component must be strongly encapsulated so that failures are fully contained and cannot spread. 6. All requests sent to a component MUST be self describing as is practical so that processing can resume with a little recovery cost as possible after a restart. 69. Let-it-Crash : Comparison Erlang Vs. Microservices Vs. Monolithic Apps 69 Erlang Philosophy Micro Services Architecture Monolithic Apps (Java, C++, C#, Node JS ...) 1 Perspective Everything is a Process Event Sourced Aggregates are a way to model the Process and NOT things. Things (defined as Objects) and Behaviors 2 Crash Recovery Supervisor will decide how to handle the crashed process Kubernetes Manager monitors all the Pods (Microservices) and its Readiness and Health. K8s terminates the Pod if the health is bad and spawns a new Pod. Circuit Breaker Pattern is used handle the fallback mechanism. Not available. Most of the monolithic Apps are Stateful and Crash Recovery needs to be handled manually and all languages other than Erlang focuses on defensive programming. 3 Concurrency Message Passing Concurrency Domain Events for state changes within a Bounded Context & Integration Events for external Systems. Mostly Shared State Concurrency 4 State Stateless : Mostly Immutable Structures Immutability is handled thru Event Sourcing along with Domain Events and Integration Events. Predominantly Stateful with Mutable structures and Mutex as a Locking Mechanism 5 Citizen Messages Messages are 1st class citizen by Event Sourcing / CQRS pattern with a strong focus on Domain Models Mutable Objects and Strong focus on Domain Models and synchronous communication. 70. Infrastructure Design Patterns Summary 4/1/2019 70 1. API Gateway 2. Service Discovery 3. Load Balancer 4. Circuit Breaker 5. Service Aggregator Pattern 6. Let It Crash Pattern 71. CAPABILITY CENTRIC DESIGN â BUSINESS FUNCTIONS â BUSINESS PROCESS â TEAM STRUCTURE 4/1/2019 71 72. Business Solution & Business Process 4/1/2019 72 - Business Solution focuses the entire Journey of the User which can run across multiple Microservices. - Business Solution comprises a set of Business Processes. - A specific Microservice functionality will be focused on a Business Process / Concern - Business Process can be divided further into Business Functions Business Solution: Customer Dining Experience Order PaymentFood Menu KitchenDining Browse Menu Order Dinner Dinner Served Get Bill Make Payment 73. 4/1/2019 73 Capability Centric Design Vertically sliced Product Team Business Centric Development â Focus on Business Capabilities â Entire team is aligned towards Business Capability. â From Specs to Operations â The team handles the entire spectrum of Software development. â Every vertical will have itâs own Code Pipeline Front-End-Team Back-End-Team Database-Team In a typical Monolithic way the team is divided based on technology / skill set rather than business functions. This leads to not only bottlenecks but also lack of understanding of the Business Domain. QA / QC Team Front-End Back-End Database Business Capability 1 QA/QCTeam Front-End Back-End Database Business Capability 2 QA/QCTeam Front-End Back-End Database Business Capability 3 QA/QCTeam 74. 74 From Project Based Activity Oriented To Product Based Outcome Oriented Source: Sriram Narayanâ https://martinfowler.com/bliki/BusinessCapabilityCentric.html 75. Capability Centric Design Summary 4/1/2019 75 1. Business Solutions 2. Business Process 3. Business Capabilities 4. Business Driven Teams (From Specs to Ops) 5. Outcome Oriented instead of Activity Oriented. 76. 4/1/2019 (C)COPYRIGHTMETAMAGICGLOBALINC.,NEWJERSEY,USA 76 Software Design Patterns â DOMAIN DRIVEN DESIGN â EVENT SOURCING AND CQRS â FUNCTIONAL REACTIVE PROGRAMMING â DISTRIBUTED TRANSACTIONS â REDUX UI PATTERN â CASE STUDY 77. 4/1/2019 77 Domain Driven Design â STRATEGIC: BOUNDED CONTEXT, UBIQUITOUS LANGUAGE â TACTICAL DESIGN: ENTITIES, AGGREGATE ROOT, VALUE OBJECT, FACTORIES, REPOSITORY, EVENTS, SERVICES â CASE STUDY: SHOPPING PORTAL 78. DDD: Bounded Context â Strategic Design 01-04-2019 78 â Bounded Context is a Specific Business Process / Concern. â Components / Modules inside the Bounded Context are context specific. â Multiple Bounded Contexts are linked using Context Mapping. â One Team assigned to a Bounded Context. â Each Bounded Context will have itâs own Source Code Repository. â When the Bounded Context is being developed as a key strategic initiative of your organization, itâs called the Core Domain. â Within a Bounded Context the team must have same language called Ubiquitous language for Spoken and for Design / Code Implementation. 79. DDD: App Userâs Journey & Bounded Context 4/1/2019 79 An App Userâs Journey can run across multiple Bounded Context / Micro Services. User Journey X Bounded Context Bounded Context Bounded Context User Journey Y Bounded Context Bounded Context Bounded Context Dinning Order Reservation Tables Recipes Raw Materials Frozen Semi Cooked Appetizer Veg Appetizer Non Veg Soft Drinks Main Course Non Veg Main Course Veg Hot Drinks Desserts Steward Chef Menu uses uses Dinning Order Reservation Tables Recipes Raw Materials Frozen Semi Cooked Appetizer Veg Appetizer Non Veg Soft Drinks Main Course Non Veg Main Course Veg Hot Drinks Desserts Steward Chef Menu uses uses UnderstandingBoundedContext(DDD)ofaRestaurantApp Dinning Context Kitchen Context Menu Context Source: Domain-Driven Design Reference by Eric Evans 80. 4/1/2019 80 Ubiquitous Language Vocabulary shared by all involved parties Used in all forms of spoken / written communication Ubiquitous Language Domain Expert Analyst Developers QA Design Docs Test Cases Code Restaurant Context â Food Item : Eg. Food Item (Navrathnakurma) can have different meaning or properties depends on the context. â In the Menu Context itâs a Veg Dish. â In the Kitchen Context itâs is recipe. â And in the Dining Context it will have more info related to user feed back etc. DDD: Ubiquitous Language: Strategic Design As an Restaurant Owner I want to know who my Customers are So that I can serve them better Role-Feature-Reason Matrix BDD â Behavior Driven Development Given Customer John Doe exists When Customer orders food Then Assign customer preferences as Veg or Non Veg customer BDD Construct 81. 01-04-2019 81 Hexagonal Architecture Ports & Adapters The layer between the Adapter and the Domain is identified as the Ports layer. The Domain is inside the port, adapters for external entities are on the outside of the port. The notion of a âportâ invokes the OS idea that any device that adheres to a known protocol can be plugged into a port. Similarly many adapters may use the Ports. Source : http://alistair.cockburn.us/Hexagonal+architecture https://skillsmatter.com/skillscasts/5744-decoupling-from-asp-net-hexagonal-architectures-in-net Services for UI Ports File system Database Order Tracking JPA Repository Implementation Adapters OrderProcessing Domain Service (Business Rules) Implementation Domain Models Domain Layer Order Data Validation OrderService REST Service Implementation OrderProcessing Interface p Order Tracking Repository Interface p A A External Apps A A A Others A A OrderService Interface p Web Services Data Store Use Case Boundary Bounded Context A â Reduces Technical Debt â Dependency Injection â Auto Wiring 82. Layered Architecture 01-04-2019 82 â Explicit Domain Models â Isolate your models from UI, Business Logic. â Domain Objects â Free of the Responsibility of displaying themselves or storing themselves or managing App Tasks. â Zero Dependency on Infrastructure, UI and Persistent Layers. â Use Dependency Injection for Loosely Coupled Objects. â All the Code for Domain Model in a Single Layer. â Domain Model should be Rich enough to represent Business Knowledge. Source: DDD Reference by Chris Evans Page 17 83. 4/1/2019 83 Domain Driven Design â Tactical Design Source: Domain-Driven Design Reference by Eric Evans 84. DDD: Understanding Aggregate Root 84 Order Customer Shipping Address Aggregate Root Line Item Line Item Line Item * Payment Strategy Credit Card Cash Bank Transfer Source: Martin Fowler : Aggregate Root â An aggregate will have one of its component objects be the aggregate root. Any references from outside the aggregate should only go to the aggregate root. The root can thus ensure the integrity of the aggregate as a whole. â Aggregates are the basic element of transfer of data storage - you request to load or save whole aggregates. Transactions should not cross aggregate boundaries. â Aggregates are sometimes confused with collection classes (lists, maps, etc.). â Aggregates are domain concepts (order, clinic visit, playlist), while collections are generic. An aggregate will often contain multiple collections, together with simple fields. 125 Domain Driven Design (C) COPYRIGHT METAMAGIC GLOBAL INC., NEW JERSEY, USA01-04-2019 85. DDD: Domain Events & Integration Events 01-04-2019 85 1. Domain Events represent something happened in a specific Domain. 2. Domain Events should be used to propagate STATE changes across Multiple Aggregates within the Bounded Context. 3. The purpose of Integration Events is to propagate committed transactions and updates to additional subsystems, whether they are other microservices, Bounded Contexts or even external applications. Source: Domain Events : Design and Implementation â Microsoft Docs â May 26, 2017 Domain Data Behavior Order (Aggregate Root) Data Behavior Address (Value Object) Data Behavior OrderItem (Child) 1 n 1 1 Order Created Domain Event Domain Layer Enforce consistency with other Aggregates Event Handler 1 Event Handler n Create and Publish Integration Event to Event Bus. Example: Order Placed Integration Event can be subscribed by Inventory system to update the Inventory details. Event Handler 2 86. 4/1/2019 86 Shopping Portal Modules â Code Packaging Auth Products Cart OrderCustomer Domain Layer â Models â Repo â Services â Factories Adapters â Repo â Services â Web Services Domain Layer â Models â Repo â Services â Factories Adapters â Repo â Services â Web Services Domain Layer â Models â Repo â Services â Factories Adapters â Repo â Services â Web Services Packaging Structure Bounded Context Implementation (Repositories, Business Services, Web Services) Domain Models (Entities, Value Objects, DTOs) (Repositories, Business Services, Web Services) Entity Factories Interfaces (Ports) 87. 4/1/2019 87 DDD: Use Case Order Module Models Value Object â Currency â Item Value â Order Status â Payment Type â Record State â Audit Log Entity â Order (Aggregate Root) â Order Item â Shipping Address â Payment DTO â Order â Order Item â Shipping Address â Payment Domain Layer Adapters â Order Repository â Order Service â Order Web Service â Order Query Web Service â Shipping Address Web Service â Payment Web Service Adapters Consists of Actual Implementation of the Ports like Database Access, Web Services API etc. Converters are used to convert an Enum value to a proper Integer value in the Database. For Example Order Status Complete is mapped to integer value 100 in the database. Services / Ports â Order Repository â Order Service â Order Web Service Utils â Order Factory â Order Status Converter â Record State Converter â Order Query Web Service â Shipping Address Web Service â Payment Web Service Shopping Portal 88. Procedural Design Vs. Domain Driven Design 88 1. Anemic Entity Structure 2. Massive IF Statements 3. Entire Logic resides in Service Layer 4. Type Dependent calculations are done based on conditional checks in Service Layer 4 1 23 Domain Driven Design with Java EE 6 By Adam Bien | Javaworld Source: http://www.javaworld.com/article/2078042/java-app-dev/domain-driven-design-with-java-ee-6.html 89. Polymorphic Business Logic inside a Domain object 01-04-2019 89 Domain Driven Design with Java EE 6 By Adam Bien | Javaworld Computation of the total cost realized inside a rich Persistent Domain Object (PDO) and not inside a service. This simplifies creating very complex business rules. Source: http://www.javaworld.com/article/2078042/java-app-dev/domain-driven-design-with-java-ee-6.html 90. Type Specific Computation in a Sub Class 90 We can change the computation of the shipping cost of a Bulky Item without touching the remaining classes. Its easy to introduce a new Sub Class without affecting the computation of the total cost in the Load Class. Domain Driven Design with Java EE 6 By Adam Bien | Javaworld of Source: http://www.javaworld.com/article/2078042/java-app-dev/domain-driven-design-with-java-ee-6.html 91. Object Construction : Procedural Way Vs. Builder Pattern 91 Procedural Way Builder Pattern Source: http://www.javaworld.com/article/2078042/java-app-dev/domain-driven-design-with-java-ee-6.html Domain Driven Design with Java EE 6 By Adam Bien | Javaworld 92. Domain Driven Design Summary 4/1/2019 92 1. Strategic Patterns 1. Bounded Context 2. Ubiquitous Language 2. Tactical Patterns 1. Aggregate Root 2. Repository 3. Entity and Value Object 4. Domain Events 93. 4/1/2019 93 Event Storming â EVENT SOURCING / CQRS â CASE STUDY: SHOPPING PORTAL â CASE STUDY: RESTAURANT APP â CASE STUDY: MOVIE BOOKING â CASE STUDY: MOVIE STREAMING 94. Mind Shift : From Object Modeling to Process Modeling 4/1/2019 94 Developers with Strong Object Modeling experience will have trouble making Events a first class citizen. â How do I start Event Sourcing? â Where do I Start on Event Sourcing / CQRS? The Key is: 1. App Userâs Journey 2. Business Process 3. Ubiquitous Language â DDD 4. Capability Centric Design 5. Outcome Oriented The Best tool to define your process and its tasks. How do you define your End Userâs Journey & Business Process? â Think It â Build It â Run IT 95. 95 Process â Define your Business Processes. Eg. Various aspects of Order Processing in an E-Commerce Site, Movie Ticket Booking, Patient visit in Hospital.1 Commands â Define the Commands (End-User interaction with your App) to execute the Process. Eg. Add Item to Cart is a Command.2 Event Sourced Aggregate â Current state of the Aggregate is always derived from the Event Store. Eg. Shopping Cart, Order etc. This will be part of the Rich Domain Model (Bounded Context) of the Micro Service.4 Projections â Projections focuses on the View perspective of the Application. As the Read & Write are different Models, you can have different Projections based on your View perspective. 5 Write Data Read Data Events â Commands generates the Events to be stored in Event Store. Eg. Item Added Event (in the Shopping Cart).3 Event Storming â Concept 96. 4/1/2019 96 Event Sourcing Intro Standard CRUD Operations â Customer Profile â Aggregate Root Profile Created Title Updated New Address added Derived Notes Removed Time T1 T2 T4T3 Event Sourcing and Derived Aggregate Root Commands 1. Create Profile 2. Update Title 3. Add Address 4. Delete Notes 2 Events 1. Profile Created Event 2. Title Updated Event 3. Address Added Event 4. Notes Deleted Event 3 Current State of the Customer Profile 4 Event store Single Source of Truth Greg Young 97. Event Sourcing & CQRS (Command and Query Responsibility Segregation) â In traditional data management systems, both commands (updates to the data) and queries (requests for data) are executed against the same set of entities in a single data repository. â CQRS is a pattern that segregates the operations that read data (Queries) from the operations that update data (Commands) by using separate interfaces. â CQRS should only be used on specific portions of a system in Bounded Context (in DDD). â CQRS should be used along with Event Sourcing. 4/1/2019 97 MSDN â Microsoft https://msdn.microsoft.com/en-us/library/dn568103.aspx | Martin Fowler : CQRS â http://martinfowler.com/bliki/CQRS.html CQS : Bertrand Meyer Axon Framework For Java Java Axon Framework Resource : http://www.axonframework.org Greg Young (C) COPYRIGHT METAMAGIC GLOBAL INC., NEW JERSEY, USA 98. 4/1/2019 98 Case Study: Restaurant Dining â Event Sourcing and CQRS Order Payment â Add Drinks â Add Food â Update Food Commands â Open Table â Add Juice â Add Soda â Add Appetizer 1 â Add Appetizer 2 â Serve Drinks â Prepare Food â Serve Food Events â Drinks Added â Food Added â Food Updated â Food Discontinued â Table Opened â Juice Added â Soda Added â Appetizer 1 Added â Appetizer 2 Added â Juice Served â Soda Served â Appetizer Served â Food Prepared â Food Served â Prepare Bill â Process Payment â Bill Prepared â Payment Processed â Payment Approved â Payment Declined â Cash Paid When people arrive at the Restaurant and take a table, a Table is opened. They may then order drinks and food. Drinks are served immediately by the table staff, however food must be cooked by a chef. Once the chef prepared the food it can then be served. Table is closed then the bill is prepared. Microservices â Dinning Order â Billable Order Customer Journey thru Dinning Processes Processes Food Menu KitchenDining â Remove Soda â Add Food 1 â Add Food 2 â Place Order â Close Table â Remove Soda â Food 1 Added â Food 2 Added â Order Placed â Table Closed ES Aggregate 3 2 4 1 99. Case Study: Shopping Site â Event Sourcing / CQRS 4/1/2019 99 Catalogue Shopping Cart Order Payment â Search Products â Add Products â Update Products Commands â Add to Cart â Remove Item â Update Quantity Customer â Process Order â Select Address â Select Delivery Mode Events â Product Added â Product Updated â Product Discontinued â Item Added â Item Removed / Discontinued â Item Updated â Order Initiated â Address Selected â Delivery Mode Selected â Order Created â Proceed for Payment â Confirm Order for Payment â Cancel Order â Payment Initiated â Order Cancelled â Order Confirmed â OTP Send â Payment Approved â Payment Declined Commands are End-User interaction with the App and based on the commands (Actions) Events are created. These Events includes both Domain Events and Integration Events. Event Sourced Aggregates will be derived using Domain Events. Each Micro Service will have its own separate Database. Depends on the scalability requirement each of the Micro Service can be scaled separately. For Example. Catalogue can be on a 50 node cluster compared to Customer Micro Service. Microservices ESA â Customer â Shop.. Cart â Order Customer Journey thru Shopping Process The purpose of this example is to demonstrate the concept of ES / CQRS thru Event Storming principles. 2 100. Case Study: Movie Booking â Event Sourcing / CQRS 4/1/2019 100 Order Payment â Search Movies â Add Movies â Update Movies Commands â Select Movie â Select Theatre / Show â Select Tickets â Process Order â Select Food â Food Removed â Skip Food â Process Order Events â Movie Added â Movie Updated â Movie Discontinued â Movie Added â Theatre / Show Added â Tickets Added â Order Initiated â Popcorn Added â Drinks Added â Popcorn Removed â Order Finalized â Proceed for Payment â Confirm Order for Payment â Cancel Order â Payment Initiated â Order Cancelled â Order Confirmed â OTP Send â Payment Approved â Payment Declined Movies Theatres Food Microservices Commands are End-User interaction with the App and based on the commands (Actions) Events are created. These Events includes both Domain Events and Integration Events. Event Sourced Aggregates will be derived using Domain Events. Each Micro Service will have its own separate Database. Depends on the scalability requirement each of the Micro Service can be scaled separately. For Example. Theatre can be on a 50 node cluster compared to Food Micro Service. ESA â Theatre â Show â Order Customer Journey thru booking Movie Ticket The purpose of this example is to demonstrate the concept of ES / CQRS thru Event Storming principles. 101. Case Study: Movie Streaming â Event Sourcing / CQRS 4/1/2019 101 Subscription Payment â Search Movies â Add Movies â Update Movies Commands â Request Streaming â Start Movie Streaming â Pause Movie Streaming â Validate Streaming License â Validate Download License Events â Movie Added â Movie Updated â Movie Discontinued â Streaming Requested â Streaming Started â Streaming Paused â Streaming Done â Streaming Request Accepted â Streaming Request Denied â Subscribe Monthly â Subscribe Annually â Monthly Subscription Added â Yearly Subscription Added â Payment Approved â Payment Declined Discovery Microservices Commands are End-User interaction with the App and based on the commands (Actions) Events are created. These Events includes both Domain Events and Integration Events. Event Sourced Aggregates will be derived using Domain Events. Each Micro Service will have its own separate Database. Depends on the scalability requirement each of the Micro Service can be scaled separately. For Example. Theatre can be on a 50 node cluster compared to Food Micro Service. ESA â Stream List â Favorite List Customer Journey thru Streaming Movie / TV Show The purpose of this example is to demonstrate the concept of ES / CQRS thru Event Storming principles. LicenseStreaming 102. Event Sourcing & CQRS Summary 4/1/2019 102 1. Process Ex. Various aspects of Order Processing in an E-Commerce Site, Movie Ticket Booking, Patient visit in Hospital. 2. Commands End-User interaction with your App) to execute the Process. Eg. Add Item to Cart is a Command. 3. Events Item Added Event (in the Shopping Cart). 4. Event Sourced Aggregate Current state of the Aggregate is always derived from the Event Store. Eg. Shopping Cart 5. Read & Write Separates Databases 103. 4/1/2019 103 Reactive Programming â BUILDING BLOCKS: OBSERVABLE, OBSERVER, SCHEDULER, OPERATOR â COMPARISON: ITERABLE (JAVA 6), STREAMS (JAVA 8), RX JAVA â CASE STUDY: MERGE STREAMS, FILTER, SORT, TAKE 104. 4/1/2019 104 Functional Reactive Programming: 4 Building Blocks of RxJava Source of Data Stream [ Sender ]Observable1 Listens for emitted values [ Receiver ]Observer2 Source: http://reactivex.io/ Schedulers3 Schedulers are used to manage and control concurrency. 1. observeOn: Thread Observable is executed 2. subscribeOn: Thread subscribe is executed 4 Operators Content Filtering Time Filtering Transformation Operators that let you Transform, Combine, Manipulate, and work with the sequence of items emitted by Observables 105. 4/1/2019 105 Comparison : Iterable / Streams / Observable 1Building Block First Class Visitor (Consumer) Serial Operations Parallel Streams (10x Speed) Still On Next, On Complete and On Error are Serial Operations Completely Asynchronous Operations Java 8 â Blocking CallJava 6 â Blocking Call Rx Java - Freedom Source Code: https://github.com/meta-magic/rxjava 106. 4/1/2019 106 Rx 2 Java Operator : Filter / Sort / FlatMap 4Building Block Objective: toSortedList() returns an Observable with a single List containing Fruits. Using FlatMap to Transform Observable to Observable Rx Example 2 SourceCodeGitHub:https://github.com/meta-magic/Rx-Java-2 â Merge â Filter â Sort â Take 107. Functional Reactive Programming Summary 4/1/2019 107 1. Observable Source of the Data Stream 2. Observer Listens to emitted values 3. Scheduler Are used to manage and control and concurrency. 4. Operators Operators that let you Transform, Combine, Manipulate, and work with the sequence of items emitted by Observables 108. 4/1/2019 108 UI DESIGN PATTERNS â TRADITIONAL PATTERNS: MVC, MVP, MVVM â FLUX DESIGN PATTERN â REDUX DESIGN PATTERNS 109. 4/1/2019 109 UI DesignPatterns MVC/ MVP/ MVVM View Controller Model Passes calls To Fire Events Manipulates â The Controller is responsible to process incoming requests. It receives input from users via the View, then process the user's data with the help of Model and passing the results back to the View. â Typically, it acts as the coordinator between the View and the Model. Model View Controller 1 * â The View Model is responsible for exposing methods, commands, and other properties that helps to maintain the state of the view, manipulate the model as the result of actions on the view, and trigger events in the view itself. â There is many-to-one relationship between View and ViewModel means many View can be mapped to one ViewModel. â Supports two-way data binding between View and ViewModel. View ViewModel Model Passes calls To Manipulates Updates Fire Events Model View ViewModel â The Presenter is responsible for handling all UI events on behalf of the view. This receive input from users via the View, then process the user's data with the help of Model and passing the results back to the View. â Unlike view and controller, view and presenter are completely decoupled from each otherâs and communicate to each otherâs by an interface. Also, presenter does not manage the incoming request traffic as controller. â Supports two-way data binding. Model View Presenter View Presenter Model Passes calls To Fire Events Manipulates Updates1 1 110. 4/1/2019 110 UI Design Patterns Flux / Redux ViewDispatcher Every action is sent to all Stores via callbacks the stores register with the Dispatcher Store Action Action 1 * Controller-Views â Listens to Store changes â Emit Actions to Dispatcher Dispatcher â Single Dispatcher per Application â Manages the Data Flow View to Model â Receives Actions and dispatch them to Stores Stores â Contains state for a Domain (Vs. Specific Component) â In Charge of modifying the Data â Inform the views when the Data is changed by emitting the Changed Event. Flux Core Concepts 1. One way Data Flow 2. No Event Chaining 3. Entire App State is resolved in store before Views Update 4. Data Manipulation ONLY happen in one place (Store). Actions â Simple JS Objects â Contains Name of the Action and Data (Payload) â Action represent something that has happened. â Has No Business Logic 111. 4/1/2019 111 UI Design Patterns Redux Actions â Simple JS Objects â Contains Name of the Action and Data (Payload) â Has NO Business Logic â Action represent something that has happened. Store â Multiple View layers can Subscribe â View layer to Dispatch actions â Single Store for the Entire Application â Data manipulation logic moves out of store to Reducers Reducer â Pure JS Functions â No External calls â Can combine multiple reducers â A function that specifies how the state changes in response to an Action. â Reducer does NOT modify the state. It returns the NEW State. Redux Core Concepts 1. One way Data Flow 2. No Dispatcher compared to Flux 3. Immutable Store Available for React & Angular View Action State Dispatcher Reducer R R R Store Middleware Middleware Middleware â Handles External calls â Multiple Middleware's can be chained. 112. UI Design Pattern Summary 4/1/2019 112 1. MVC 2. MVP 3. MVVM 4. Flux 5. Redux Redux is a much better pattern if you are building complex enterprise applications. 113. 4/1/2019 113 Distributed Transactions â 2 PHASE COMMIT â SAGA DESIGN PATTERN â HANDLING INVARIANTS â FORWARD RECOVERY 114. Distributed Transactions : 2 Phase Commit 2 PC or not 2 PC, Wherefore Art Thou XA? 01April2019 114 How does 2PC impact scalability? â Transactions are committed in two phases. â This involves communicating with every database (XA Resources) involved to determine if the transaction will commit in the first phase. â During the second pha https://www.friendbookmark.com/videos/975/docker-containers-for-continuous-deliveryLooking for continuous delivery options that are also cloud agnostic? Docker could be your answer. This webinar answers some of your basic questions about Docker and gives you an overview of how to setup Docker for Continuous Delivery. We will also touch upon more advanced topics like Cloud Portability, Microservices deployment and MEAN stack enablement. Get details at https://www.synerzip.com/webinar/docker-containers-for-continuous-delivery-webinar-february-2015/ Topics covered: 1. Webinar Series â February 18, 2015 2. SPEAKER INTRODUCTION www.synerzip.com 2 3. Rohit Ghatol 3 â Director of Engineering @ Synerzip â Technology Evangelist Contact Details Twitter: @rohitghatol Linkedin: www.linkedin.com/in/rohitghatol HomePage: http://rohitghatol.com www.synerzip.com 4. DOCKER INTRODUCTION 4 www.synerzip.com 5. Docker As a Technology â â Light Weight Container Technology for Virtualization â Fastest Growing Technology (2 Years) â First Release March 2013 â 0.9 Release in March 2014 â Colossal Adoption Rate â 18,876 Star on Github 5 www.synerzip.com 6. Docker As a Company - â Incubated @ DotCloud (PAAS Company) â Changed to Docker Oct 2013 â Privately Held Company â $55 Million in 2 Rounds from 6 Investors â Sold DotCloud to CloudControl Aug 2014 6 www.synerzip.com 7. WHO USES DOCKER? 7 www.synerzip.com 8. Who uses Docker? Companies using Docker 8 And many moreâ www.synerzip.com 9. Who uses Docker? Docker PAAS Providers 9 And many moreâ www.synerzip.com 10. Who uses Docker? 10 As an Infrastructure Tool along side www.synerzip.com 11. CONTINUOUS DELIVERY www.synerzip.com 11 12. Continuous Delivery 12 www.synerzip.com 13. The Components 13 1.5 Host Environment 2.1 App config DB www.synerzip.com 14. Continuous Delivery 14 App confi g DB 1.0 Env 1.2 1.3 Env 2.3 1.5 Env 2.1 App DB App DB confi g confi g www.synerzip.com 15. The Components 15 Jenkins, Bamboo, etc 1.5 Host Environment 2.1 App config DB Vagrant, Puppet, Chef etc. Virtual Machines, Instructions, Commands, Etc. www.synerzip.com 16. CONTINUOUS DELIVERY THE NEXT STEPâ 16 www.synerzip.com 17. Containers 17 1.5 Env 2.1 App config DB www.synerzip.com 18. LANDSCAPE How companies are deploying SAAS today? 18 www.synerzip.com 19. Landscape 19 Dev Box Test Servers CI Staging Area Production Area QA Box Dev QA Op s DevOps Tools - Chef, Puppet, Anisble, SaltStack, Vagrant, VirtualBox, VMWare CIT Tools â Jenkins,Bamboo,Travis, etc IAAS/PAAS â AWS, Azure, Google Cloud, Digital Ocean,Heroku etc www.synerzip.com 20. DOCKER 101 20 www.synerzip.com 21. What is Docker? Docker provides Docker Daemon to run light weight containers on Linux. Applications: â Are dockerized and â Run on Docker Containers â from laptops to production servers on cloud â Their images are shared on Docker Hub â Apps can be linked (node -> mongo) 21 www.synerzip.com 22. What is a Container? 22 Virtual Machine Container Using namespace, cgroups, apparmour, etc. www.synerzip.com 23. What is Docker? 23 Docker Client Docker Daemon Public Repo Private Repo Docker File DSL www.synerzip.com 24. Docker File System 24 www.synerzip.com 25. Dockerfile FROM dockerfile/node - Base Image RUN apt-get update âqq - Instructions RUN mkdir /my/app - while building image ADD . /my/app CMD [ânodeâ,âwebâ] -What Command to run 25 www.synerzip.com 26. Dockerfile //Build an Image $> docker build ât ârohitghatol/nodeâ . //Run an Image $>docker run âd âp 80:3000 rohitghatol/node //Push to Docker Hub $>docker push rohitghatol/node //developer $>docker pull rohitghatol/node //operations 26 www.synerzip.com 27. DOCKER USE CASES 27 www.synerzip.com 28. CONTINUOUS DELIVERY 28 www.synerzip.com 29. Continuous Delivery 29 Continuous Delivery Operations Dev CI QA Staging Prod High Availability Redundancy SLAs Area where Docker shines Kubernetes etc www.synerzip.com 30. Continuous Delivery Use case 30 Dev Box Test Servers CIT Staging Area Production Area QA Box Dev QA Op s SUT Test Server V 2.1 V 2.2 V 2.3 Continuous delivery Rollbacks www.synerzip.com 31. Developer Scenario 31 Dev Box Dev Rails Image Docker Hub Web (RoR) Dev 1. Pull 2. Run 3. Customize 4. Push as Base Image Team 5. Pull www.synerzip.com 32. Developer Scenario 32 Dev Box Dev Host Machine (Developer codes here) Docker Web (RoR) DB (MySQL) ~/app/src ./src IDE, Browsers, etc => Docker Hub Sync Folder Code runs here www.synerzip.com 33. CI Scenarios 33 Code Push Test Code Test Feature Build App Docker Image Run Docker Container with App Image Publish App Docker Image Numerous combinationsâ Pull Code www.synerzip.com 34. CI Scenarios â Option 1 34 Code Push Pull Code Test Code Publish App Docker Image Build App Docker Image www.synerzip.com 35. CI Scenario 35 Github Docker Web (RoR) DB (MySQL) Docker Hub Dev Box Dev 1. Push Code 2. Listen 3. Get Base Image 5. Run Tests 4. Pull Code 6. Build App Image 7. Push App Image www.synerzip.com CI Server ( Drone, Shippable, Circle CI, CodeShip, Travis CI, Jenkins etc) 36. CI Scenarios â Option 2 36 Code Push Test Feature Build App Docker Image Run Docker Container with App Image Publish App Docker Image Mark Good App Image www.synerzip.com 37. CI Scenario 37 Github CI Server (Drone.io, inhouseCI, etc) Docker Web (RoR) DB (MySQL) Docker Hub Dev Box Dev 1. Push Code 3. Listen 2. Create App Docker Image 5. Run Tests 4. Pull Run App Image 6. Accept/Reject Image www.synerzip.com 38. Staging/Prod Scenario 38 Host Machine Docker Container Web (RoR) DB (MySQL) Docker Hub 1. Trigger Event 3. Pull App Image 4. Run Image 2. Deploy Docker Container (AWS, Azure, Digital Ocean, etc.) Host Machine www.synerzip.com 39. CLOUD PORTABILITY 39 www.synerzip.com 40. Docker Hub Cloud Portability Use case 40 Amazon AWS Google Cloud Microsoft Azure Digital Ocean Deployment Tool Docker, Swarm, Drone.io, Flocker, Tutum, etc SAAS Company Github www.synerzip.com 41. MEAN STACK 41 www.synerzip.com 42. MeanStack Use case 42 Open-Source Full-Stack Solution for MEAN Applications www.synerzip.com 43. MeanStack Use case â Mean.js provides â Code generator to generate Mean App â Mean.js apps typically have â Node Js Server â Mongo DB database â Provides Dockerfile and fig.yml to run the app in Docker Containers â One Docker container for Node Js Server â One Docker container for Mongo DB Database 43 www.synerzip.com 44. MeanStack Use case 44 www.synerzip.com 45. MeanStack Use case 45 Web DB Port 3000 Port 27017 Docker Containers fig.yml www.synerzip.com 46. MeanStack Use case 46 dockerfile/nodejs Image grunt-cli bower package.json â.. Dockerfile www.synerzip.com 47. MICRO SERVICES 47 www.synerzip.com 48. Micro Services Use case 48 Micro Service Recomm Engine Node Mong o Solr Recomm Engine Recomm Engine Movie Listing Play PSql Movie Listing Movie Listing Movie Listing Profile RoR My SQL Redis Profile Preference RoR Redis Preference www.synerzip.com 49. âââ âââ Micro Services Use case 49 Recomm Engine Movie Listing Profile Preference Sprint 1 Sprint 2 Sprint 7 Sprint 1 Sprint 2 Sprint 1 Sprint 2 Sprint 3 Sprint 1 Sprint 2 Sprint 3 âââ âââ Deploy www.synerzip.com 50. Micro Services Use case 50 Recomm Engine Node Mong o Solr Recomm Engine Recomm Engine Docker Container Gateway/Re v Proxy www.synerzip.com 51. Micro Services Use case â Micro services are hard to run â Needs strong DevOps process â Docker helps by â Defining container/micro service as unit â Shipping one micro service as one container â More containers = more scale â By improving Dev â Operations relationships 51 www.synerzip.com 52. Micro Services Use case â What else is needed? â Scheduling â High Availability â Service Discovery â Etc. 52 Giant Swarm www.synerzip.com 53. FUTURE OF DOCKER 53 www.synerzip.com 54. NEW DOCKER PRODUCTS 54 www.synerzip.com 55. New Docker Products â Docker Machine â Docker Swarm â Docker Compose 55 www.synerzip.com 56. Docker Machine â Machine makes it really easy to create Docker hosts on local hypervisors and cloud providers. â It creates servers, installs Docker on them, then configures the Docker client to talk to them. 56 www.synerzip.com 57. Docker Machine $>docker-machine create -d virtualbox dev $>docker-machine create -d digitalocean -- digitalocean-access-token=... staging 57 www.synerzip.com 58. Docker Swarm â Swarm is a simple tool which controls a cluster of Docker hosts and exposes it as a single "virtual" host. â Swarm uses the standard Docker API as its frontend, which means any tool which speaks Docker can control swarm transparently. 58 www.synerzip.com 59. Docker Swarm â # create a cluster $ docker run --rm swarm create â # Add Machine to cluster docker run -d swarm join --addr= token:// â #Run Docker commands on swarm docker -H tcp:// run âd âP dockerfile/node 59 www.synerzip.com 60. Docker Compose â An orchestration tool for Docker â Defines â Which Docker containers are to be run â How they are connected â What ports they expose â All in single file â Initial design based on Fig.sh â Current status - limbo 60 www.synerzip.com 61. IAAS/PAAS ADOPTION 61 www.synerzip.com 62. IAAS/PAAS Adoption â Amazon ECS â Container service â Supports tasks configuration â Google Cloud â Based on Kubernetes â Microsoft Azure 62 www.synerzip.com 63. Q & A? 63 www.synerzip.com 64. â6484 www.synerzip.com Ashish Shanker ashish.shanker@synerzip.com 469.374.0500 64 www.synerzip.com 65. Synerzip in a Nutshell â Software product development partner for small/mid-sized technology companies â Exclusive focus on small/mid-sized technology companies, typically venture- backed companies in growth phase â By definition, all Synerzip work is the IP of its respective clients â Deep experience in full SDLC â design, dev, QA/testing, deployment â Dedicated team of high caliber software professionals for each client â Seamlessly extends clientâs local team, offering full transparency â Stable teams with very low turn-over â NOT just âstaff augmentationâ, but provide full mgmt support â Actually reduces risk of development/delivery â Experienced team - uses appropriate level of engineering discipline â Practices Agile development â responsive, yet disciplined â Reduces cost â dual-shore team, 50% cost advantage â Offers long term flexibility â allows (facilitates) taking offshore team captive â aka âBOTâ option 65 www.synerzip.com 66. Our Clients 66 www.synerzip.com 67. Next Webinar Life Cycle of a User Story Complimentary Webinar: Tuesday, March 17, 2015 @ noon CST Presented by: Michael Hall, CEO and Founder of Three Beacons. Michael is a software practitioner and team leader, Certified Scrum Master, Certified Scrum Product Owner, and an early adopter of Agile methods since 2001. Three Beacons is a leading provider of agile training and consulting services. See www.threebeacons.com for a complete description of Agile training courses available. 67 www.synerzip.com 68. Ashish Shanker ashish.shanker@synerzip.com 469.374.0500 Thanks! @Synerzip_Agile linkedin.com/company/synerzip facebook.com/Synerzip 68 www.synerzip.com https://www.friendbookmark.com/videos/974/shipping-applications-to-production-in-containers-with-dockerDocker is an Open Source engine to build, run, and manage Linux Containers. Containers use less resources than virtual machines, they boot faster, but they have similar guarantees of portability and repeatability for Linux applications. Those features made Docker and Linux Containers extremely popular for development and testing environments. But what does it take to use Docker and Containers for production workloads? Topics covered: 1. Docker in production JÃrÃme Petazzoni Docker Inc. @jpetazzo @docker 2. JÃrÃme Petazzoni (@jpetazzo) -Grumpy French DevOps - Go away or I will replace you with a very small shell script -Wrote dotCloud PAAS deployment tools - EC2, LXC, Puppet, Python, Shell, ÃMQ... -Docker contributor - Security, networking... -Runs all kinds of crazy things in Docker - Docker-in-Docker, VPN-in-Docker, KVM-in-Docker, Xorg-in-Docker... 3. Outline -Quick recap on Docker and its 1.0 release -âSolvedâ problems: install, build, distribute -Service discovery & general plumbing -Orchestration (running many containers) -Performance (measuring it & improving it) -Configuration management -Sysadmin chores: logging, backups, remote access 4. One-slide elevator pitch about Docker -Docker is an Open Source engine for containers - build, ship, run your applications within containers (=lightweight VMs) -Docker enables separation of concerns - devs put their apps in containers - ops run the containers -It's (probably) one of the most active FOSS projects today - more than 500 contributors in the last year - includes major contributions from e.g. Google, Red Hat... 5. Docker 1.0 1.1 1.1.1 is here! -Docker 1.0 released last month for DockerCon -Random pick of recent features: - pause/unpause (helps to get consistent commit/snapshot) - SELinux (for, you know, security) - network superpowers with docker run --net â -More importantly: it's stamped âproduction-readyâ - you can buy support contracts, training... (in addition to the traditional t-shirts and stickers )☺ 6. Installation -On your dev machine: boot2docker - tiny VM (25 MB), works with all virtualization types - wrapper script (OS X only) to run docker CLI locally - future improvements: shared volumes with docker run -v â -On your servers: which distro? - use something recent (Ubuntu 14.04 LTS, RHEL 7, Fedora 20...) - special distros: CoreOS, Project Atomic â new but promising 7. Build with Dockerfiles FROM ubuntu:14.04 MAINTAINER Docker Education Team RUN apt-get update RUN apt-get install -y nginx RUN echo 'Hi, I am in your container' >/usr/share/nginx/html/index.html CMD [ "nginx", "-g", "daemon off;" ] EXPOSE 80 8. Build with Dockerfiles -Great for most purposes - caching system allows full rebuilds that are still fast -Drawbacks (a.k.a. work in progress) - separate build/run environments (don't ship that 5 GB build image if you just need the 10 MB artifact) - entitlement, credentials, and other secrets (what if the build process needs to access a private repository?) -Workarounds - use two Dockerfiles; keep Dockerfiles and images private 9. Distribute and ship images -Docker Hub - docker push, docker pull: it's magic! - public and private images - no on prem version yet; but it's one of the most requested features -Run your own registry - docker run registry # âdocker run -Pâ to expose it to LAN - defaults to local storage - can use cloud object storage (Swift, GCE, S3, Elliptics...) 10. Distribute and ship images -Hack around docker load/save - load/save works with plain tarballs - put them wherever you want them - https://github.com/blake-education/dogestry (much image, such docker, wow) -Work in progress: pluggable transports - many things are damn good at moving diffs (git, rsync...) - can we borrow something from them? 11. Service discovery -There's more than one way to do it - inject everything we need through environment docker run -e DB_HOST=â -e DB_PORT=â -e â - bind-mount a configuration file into the container docker run -v /etc/docker/config/myapp.yaml:/config.yaml â - resolve everything we need through a highly-available key-value store (zookeeper, etcd, consul...) - resolve everything we need through DNS (consul, skydns, skydock, dnsmasq...) 12. How do they compare? Let's grade those different methods! 13. But first, let's look at links 14. Docker links docker run -d --name frontdb mysqlimage docker run -d --link frontdb:sql webimage -DNS entries are created in containers -Environment variables are injected in 2nd container SQL_PORT=tcp://172.17.0.10:5432 SQL_PORT_5432_TCP=tcp://172.17.0.10:5432 SQL_PORT_5432_TCP_ADDR=172.17.0.10 SQL_PORT_5432_TCP_PORT=5432 SQL_PORT_5432_TCP_PROTO=tcp -Doesn't work across multiple Docker hosts 15. Service discovery: environment variables -Easy to integrate in your code - is there any language that does not support environment variables? -Easy to setup - start services, lookup ports, inject variables -Even easier with links - fully automatic if using only one host -Static - if a service moves, cannot update environment variables 16. Environment variables: B 17. Service discovery: bind-mount configuration file -Easy to integrate in your code - again, is there a language without a decent JSON/YAML parser? -Easy to setup - just like environment variables, but generate a file -Kind of dynamic - it's possible to update the configuration files while services run -But not really - services have to detect the change and reload the file 18. Bind-mount configuration file: B 19. Service discovery: key-value store -Harder to integrate in your code - HTTP requests instead of getenv are not too hard, but still -Harder to setup - must setup the key-value store; on multiple nodes -Kind of dynamic - most of those key-value stores support âwatchâ operation -But not really - services still have to detect the change and reload the file 20. Key-value stores: D 21. Service discovery: DNS -Easy to integrate in your code - in most cases, no integration is needed at all, works out of the box -Harder to setup* - must setup a DNS system that you can easily update -Dynamic - you can update DNS zones, no problem -No âpushâ, but... - services won't detect a change, but if something wrong happens (and results into a disconnection) they might re-resolve and retry *Except on a single host, if you use links, since they automatically create DNS entries. 22. DNS: B 23. Are we doomed? 24. Links, take two 25. The ambassador pattern host 1 (database) docker run -d -name frontdb mysqlimage docker run -d -link frontdb:sql wiring host 2 (web tier) docker run -d -name frontdb wiring docker run -d -link frontdb:sql nginximage 26. database host web host database container I'm frontdb! web container I want to talk to frontdb! wiring container I actually talk to frontdb! wiring container I pretend I'm frontdb! docker link docker link ? 27. database host web host database container I'm frontdb! web container I want to talk to frontdb! wiring container I actually talk to frontdb! wiring container I pretend I'm frontdb! docker link docker link ? 28. database host web host database container I'm frontdb! web container I want to talk to frontdb! wiring container I actually talk to frontdb! wiring container I pretend I'm frontdb! docker link docker link UNICORNS 29. â...Unicorns?â -Work in progress, but you can look at: - Docksul https://github.com/progrium/docksul - Grand Ambassador https://github.com/cpuguy83/docker-grand-ambassador -Or roll your own - use some highly-available key-value store (yup, they're back too!) - HAProxy, stunnel, iptables... 30. Service discovery: links with ambassadors -Easy to integrate in your code - it's still environment variables -Easy to setup in dev, harder in production - use normal links in dev; get the big guns out only in prod -Dynamic - the ambassadors can reroute traffic if necessary 31. Ambassadors: A 32. But warning: construction area (They're still work in progress) 33. Orchestration -There's more than one way to do it (again!) - describe your stack in files (Fig, Maestro-NG, Ansible and other CMs) - submit requests through an API (Mesos) - implement something that looks like a PAAS (Flynn, Deis, OpenShift) - the ânew waveâ (Kubernetes, Centurion, Helios...) - OpenStack (because OpenStack can do everything!) 34. Introducing the Docker orchestration flowchart 35. Do you (want to) use OpenStack? -Yes - if you are building a PAAS, keep an eye on Solum (and consider contributing) - if you are moving VM workloads to containers, use Nova (that's probably what you already have; just enable the Docker driver) - otherwise, use Heat (and use Docker resources in your Heat templates) -No - go to next slide 36. Are you looking for a PAAS? -Yes - CloudFoundry (Ruby, but increasing % Go) - Deis (Python, Docker-ish, runs on top of CoreOS) - Dokku (A few 100s of line of Bash!) - Flynn (Go, bleeding edge) - OpenShift geard (Go) -Choose wisely (or go to the next slide) - http://blog.lusis.org/blog/2014/06/14/paas-for-realists/ âI donât think ANY of the current private PaaS solutions are a fit right now.â 37. How many Docker hosts do you have? -Only one per app or environment - Fig -A few (up to ~10) - Maestro-NG - your favorite CM (e.g. Ansible has a nice Docker module) -A lot - Mesos - have a look at (and contribute to) the ânew waveâ (Centurion, Helios, Kubernetes...) 38. Work in progress: libswarm -Run that... - exposes the Docker API - talks to real Docker hosts - spins Docker hosts up and down as needed - takes care of scheduling, plumbing, scaling... -Use your normal client to talk to that - it looks like a Docker host - but it's an elastic, scalable, dynamic, magic Docker host -https://github.com/docker/libswarm 39. Performance: measure things -cgroups give us per-container... - CPU usage - memory usage (fine-grained: cache and resident set size) - I/O usage (per device, reads vs writes, in bytes and in ops) -cgroups don't give us... - network metrics (have to do tricks with network namespaces) https://github.com/google/cadvisor http://jpetazzo.github.io/2013/10/08/docker-containers-metrics/ 40. Performance: tweak things -There isn't much to tweak! - CPU: native - I/O: native on volumes (make sure that your data set etc. is on volumes) - memory: no overhead if you disable memory accounting (userful for HPC; probably not for everything else) - network: no overhead if you run with â--net hostâ (useful for >1 Gb/s workloads) (or if you have a high packet rate; e.g. VOIP, gaming...) 41. Configuration management -There is more than one way do to it (surprise!) -If you don't use a CM system yet, you don't have to - If you're familiar with a CM system, you can use it to encode small- scale deployments (up to, say, 10 nodes) -Using CM to manage Docker hosts makes sense -But Dockerfiles will be great for apps themselves -If you really want to keep using your recipes, here's how to integrate! 42. Configuration management, if you want to mix VMs and containers -Author a single generic Docker image with your favorite CM, âlocked and loadedâ -When creating a container from that image, you give it its identity (certificate/node name/...) -When the container starts, it contacts the server, which gives it its configuration (manifests, cookbooks...) -After a moment, it will converge to desired state -Downside: slow to converge; not 100% reliable 43. Configuration management, if you want to mix VMs and containers -Author a single generic Docker image with your favorite CM, âlocked and loadedâ -When creating a container from that image, you give it its identity (certificate/node name/...) -When the container starts, it contacts the server, which gives it its configuration (manifests, cookbooks...) -After a moment, it will converge to desired state -Downside: slow to converge; not 100% reliable NOT RECOMMENDED 44. Configuration management, the âimmutable infrastructureâ way -Author a single generic Docker image with your favorite CM, to be used as a base for other images -Author other Docker images: FROM me/my_base_puppet_image ADD manifests/ /etc/puppet/manifests RUN puppet apply --certname db1138.dystopia.io -Once the image is baked, you don't have to fry it (i.e. it's ready to run without extra steps) -Downside: build new image to make a change (can be seen as an advantage) 45. Configuration management, the âimmutable infrastructureâ way -Author a single generic Docker image with your favorite CM, to be used as a base for other images -Author other Docker images: FROM me/my_base_puppet_image ADD manifests/ /etc/puppet/manifests RUN puppet apply --certname db1138.dystopia.io -Once the image is baked, you don't have to fry it (i.e. it's ready to run without extra steps) -Downside: build new image to make a change (can be seen as an advantage) SLIGHTLY BETTER (BUT STILL KIND OF MEH) 46. Sysadmin chores -Backups -Logging -Remote access We all know that those are just a small sample of the many boring, necessary evil deeds that sysadmins must commit once in a while. 47. File-level backups -Use volumes docker run --name mysqldata -v /var/lib/mysql busybox true docker run --name mysql --volumes-from mysqldata mysql docker run --rm --volumes-from mysqldata mysqlbackup tar -cJf- /var/lib/mysql | stream-it-to-the-cloud.py -Of course, you can use anything fancier than tar (e.g. rsync, tarsnap...) 48. Data-level backups -Use links docker run --name mysql mysql docker run --rm --link mysql:db mysqlbackup mysqldump --all-databases | stream-it-to-the-cloud.py -Can be combined with volumes - put the SQL dump on a volume - then backup that volume with file-level tools (previous slide) 49. Logging for legacy apps -Legacy = let me write to eleventy jillion arbitrary files in /var/lib/tomcat/logs! -Solution: volumes docker run --name logs -v /var/lib/tomcat/logs busybox true docker run --name tomcat --volumes-from logs my_tomcat_image - Inspect logs: docker run --rm --volumes-from logs ubuntu bash - Ship logs to something else: docker run --name logshipper --volumes-from logs sawmill 50. Logging for dockerized apps -Dockerized = I only write to stdout -Solution: Docker CLI/API docker run --name tomcat dockerized_tomcat docker logs tomcat docker run -v /var/run/docker.sock:/var/run/docker.sock logshipper docker logs tomcat | pipestash ... -Caveat: logs are not rotated (but PR is on the way) 51. Remote access -If you own the host: SSH to host + nsenter https://github.com/jpetazzo/nsenter -If you don't own the host: SSH in the container https://github.com/phusion/baseimage-docker -More on that topic (âdo I need SSHD in containers?â): http://blog.docker.com/2014/06/why-you-dont-need-to-run-sshd-in-docker/ -In the future: - run separate SSH container - log into that - âhopâ onto the target container 52. Docker in production Containers, containers everywhere! Not an actual book (yet) Thank you! Questions? http://www.docker.com/ @docker @jpetazzo Come talk about Docker tomorrow: - 10:40am: office hours (expo hall table A) - evening: meet-up at New Relic https://www.friendbookmark.com/videos/973/an-intro-to-docker-terraform-and-amazon-ecsThis talk is a very quick intro to Docker, Terraform, and Amazon's EC2 Container Service (ECS). In just 15 minutes, you'll see how to take two apps (a Rails frontend and a Sinatra backend), package them as Docker containers, run them using Amazon ECS, and to define all of the infrastructure-as-code using Terraform. Topics covered: 1. A quick intro to Docker, Terraform, and Amazon ECS TERRAFORM Amazon ECS 2. In this talk, weâll show how to deploy two apps: 3. A Rails Frontend and a Sinatra Backend 4. Slides and code from this talk: ybrikman.com/speaking 5. require 'sinatra' get "/" do "Hello, World!" end The sinatra backend just returns âHello, Worldâ. 6. class ApplicationController < ActionController::Base def index url = URI.parse(backend_addr) req = Net::HTTP::Get.new(url.to_s) res = Net::HTTP.start(url.host, url.port) {|http| http.request(req) } @text = res.body end end The rails frontend calls the sinatra backendâ 7. Rails Frontend Response from the backend: And renders the response as HTML. 8. Weâll package the two apps as Docker containersâ 9. Amazon ECS Deploy those Docker containers using Amazon ECSâ 10. TERRAFORM And define our infrastructure-as- code using Terraform. 11. Iâm Yevgeniy Brikman ybrikman.com 12. Co-founder of Gruntwork gruntwork.io 13. gruntwork.io We offer DevOps as a Service 14. gruntwork.io And DevOps as a Library 15. PAST LIVES 16. Author of Hello, Startup hello-startup.net 17. And Terraform: Up & Running terraformupandrunning.com 18. 1. Docker 2. Terraform 3. ECS 4. Recap Outline 19. 1. Docker 2. Terraform 3. ECS 4. Recap Outline 20. Docker allows you to build and run code in containers 21. Containers are like lightweight Virtual Machines (VMs) 22. Like an isolated process that happens to be an entire OS 23. > docker run âit ubuntu bash root@12345:/# echo "I'm in $(cat /etc/issue)â I'm in Ubuntu 14.04.4 LTS Running an Ubuntu image in a Docker container 24. > time docker run ubuntu echo "Hello, World" Hello, World real 0m0.183s user 0m0.009s sys 0m0.014s Containers boot quickly, with minimal CPU/memory overhead 25. You can define a Docker image as code in a Dockerfile 26. FROM gliderlabs/alpine:3.3 RUN apk --no-cache add ruby ruby-dev RUN gem install sinatra --no-ri --no-rdoc RUN mkdir -p /usr/src/app COPY . /usr/src/app WORKDIR /usr/src/app EXPOSE 4567 CMD ["ruby", "app.rb"] Here is the Dockerfile for the Sinatra backend 27. FROM gliderlabs/alpine:3.3 RUN apk --no-cache add ruby ruby-dev RUN gem install sinatra --no-ri --no-rdoc RUN mkdir -p /usr/src/app COPY . /usr/src/app WORKDIR /usr/src/app EXPOSE 4567 CMD ["ruby", "app.rb"] It specifies dependencies, code, config, and how to run the app 28. > docker build -t gruntwork/sinatra-backend . Step 0 : FROM gliderlabs/alpine:3.3 ---> 0a7e169bce21 (...) Step 8 : CMD ruby app.rb ---> 2e243eba30ed Successfully built 2e243eba30ed Build the Docker image 29. > docker run -it -p 4567:4567 gruntwork/sinatra-backend INFO WEBrick 1.3.1 INFO ruby 2.2.4 (2015-12-16) [x86_64-linux-musl] == Sinatra (v1.4.7) has taken the stage on 4567 for development with backup from WEBrick INFO WEBrick::HTTPServer#start: pid=1 port=4567 Run the Docker image 30. > docker push gruntwork/sinatra-backend The push refers to a repository [docker.io/gruntwork/sinatra- backend] (len: 1) 2e243eba30ed: Image successfully pushed 7e2e0c53e246: Image successfully pushed 919d9a73b500: Image successfully pushed (...) v1: digest: sha256:09f48ed773966ec7fe4558 size: 14319 You can share your images by pushing them to Docker Hub 31. Now you can reuse the same image in dev, stg, prod, etc 32. > docker pull rails:4.2.6 And you can reuse images created by others. 33. FROM rails:4.2.6 RUN mkdir -p /usr/src/app COPY . /usr/src/app WORKDIR /usr/src/app RUN bundle install EXPOSE 3000 CMD ["rails", "start"] The rails-frontend is built on top of the official rails Docker image 34. rails_frontend: image: gruntwork/rails-frontend ports: - "3000:3000" links: - sinatra_backend sinatra_backend: image: gruntwork/sinatra-backend ports: - "4567:4567" Define your entire dev stack as code with docker-compose 35. rails_frontend: image: gruntwork/rails-frontend ports: - "3000:3000" links: - sinatra_backend sinatra_backend: image: gruntwork/sinatra-backend ports: - "4567:4567" Docker links provide a simple service discovery mechanism 36. > docker-compose up Starting infrastructureascodetalk_sinatra_backend_1 Recreating infrastructureascodetalk_rails_frontend_1 sinatra_backend_1 | INFO WEBrick 1.3.1 sinatra_backend_1 | INFO ruby 2.2.4 (2015-12-16) sinatra_backend_1 | Sinatra has taken the stage on 4567 rails_frontend_1 | INFO WEBrick 1.3.1 rails_frontend_1 | INFO ruby 2.3.0 (2015-12-25) rails_frontend_1 | INFO WEBrick::HTTPServer#start: port=3000 Run your entire dev stack with one command 37. 1. Docker 2. Terraform 3. ECS 4. Recap Outline 38. Terraform is a tool for provisioning infrastructure 39. Terraform supports many providers (cloud agnostic) 40. And many resources for each provider 41. You define infrastructure as code in Terraform templates 42. provider "aws" { region = "us-east-1" } resource "aws_instance" "example" { ami = "ami-408c7f28" instance_type = "t2.micro" } This template creates a single EC2 instance in AWS 43. > terraform plan + aws_instance.example ami: "" => "ami-408c7f28" instance_type: "" => "t2.micro" key_name: "" => "" private_ip: "" => "" public_ip: "" => "" Plan: 1 to add, 0 to change, 0 to destroy. Use the plan command to see what youâre about to deploy 44. > terraform apply aws_instance.example: Creating... ami: "" => "ami-408c7f28" instance_type: "" => "t2.micro" key_name: "" => "" private_ip: "" => "" public_ip: "" => "â aws_instance.example: Creation complete Apply complete! Resources: 1 added, 0 changed, 0 destroyed. Use the apply command to apply the changes 45. Now our EC2 instance is running! 46. resource "aws_instance" "example" { ami = "ami-408c7f28" instance_type = "t2.micro" tags { Name = "terraform-example" } } Letâs give the EC2 instance a tag with a readable name 47. > terraform plan ~ aws_instance.example tags.#: "0" => "1" tags.Name: "" => "terraform-example" Plan: 0 to add, 1 to change, 0 to destroy. Use the plan command again to verify your changes 48. > terraform apply aws_instance.example: Refreshing state... aws_instance.example: Modifying... tags.#: "0" => "1" tags.Name: "" => "terraform-example" aws_instance.example: Modifications complete Apply complete! Resources: 0 added, 1 changed, 0 destroyed. Use the apply command again to deploy those changes 49. Now our EC2 instance has a tag! 50. resource "aws_elb" "example" { name = "example" availability_zones = ["us-east-1a", "us-east-1b"] instances = ["${aws_instance.example.id}"] listener { lb_port = 80 lb_protocol = "http" instance_port = "${var.instance_port}" instance_protocol = "httpâ } } Letâs add an Elastic Load Balancer (ELB). 51. resource "aws_elb" "example" { name = "example" availability_zones = ["us-east-1a", "us-east-1b"] instances = ["${aws_instance.example.id}"] listener { lb_port = 80 lb_protocol = "http" instance_port = "${var.instance_port}" instance_protocol = "httpâ } } Terraform supports variables, such as var.instance_port 52. resource "aws_elb" "example" { name = "example" availability_zones = ["us-east-1a", "us-east-1b"] instances = ["${aws_instance.example.id}"] listener { lb_port = 80 lb_protocol = "http" instance_port = "${var.instance_port}" instance_protocol = "http" } } As well as dependencies like aws_instance.example.id 53. resource "aws_elb" "example" { name = "example" availability_zones = ["us-east-1a", "us-east-1b"] instances = ["${aws_instance.example.id}"] listener { lb_port = 80 lb_protocol = "http" instance_port = "${var.instance_port}" instance_protocol = "http" } } It builds a dependency graph and applies it in parallel. 54. After running apply, we have an ELB! 55. > terraform destroy aws_instance.example: Refreshing state... (ID: i-f3d58c70) aws_elb.example: Refreshing state... (ID: example) aws_elb.example: Destroying... aws_elb.example: Destruction complete aws_instance.example: Destroying... aws_instance.example: Destruction complete Apply complete! Resources: 0 added, 0 changed, 2 destroyed. Use the destroy command to delete all your resources 56. For more info, check out The Comprehensive Guide to Terraform 57. 1. Docker 2. Terraform 3. ECS 4. Recap Outline 58. EC2 Container Service (ECS) is a way to run Docker on AWS 59. ECS Overview EC2 Instance ECS Cluster ECS Scheduler ECS Agent ECS Tasks ECS Task Definition { "cluster": "example", "serviceName": âfoo", "taskDefinition": "", "desiredCount": 2 } ECS Service Definition { "name": "example", "image": "foo/example", "cpu": 1024, "memory": 2048, "essential": true, } 60. ECS Cluster: several servers managed by ECS EC2 Instance ECS Cluster 61. Typically, the servers are in an Auto Scaling Group Auto Scaling Group EC2 Instance 62. Each server must run the ECS Agent ECS Agent EC2 Instance ECS Cluster 63. ECS Task: Docker container(s) to run, resources they need { "name": "example", "image": "foo/example", "cpu": 1024, "memory": 2048, "essential": true, } ECS Agent EC2 Instance ECS Task Definition ECS Cluster 64. ECS Service: long-running ECS Task & ELB settings { "name": "example", "image": "foo/example", "cpu": 1024, "memory": 2048, "essential": true, } { "cluster": "example", "serviceName": âfoo", "taskDefinition": "", "desiredCount": 2 } ECS Agent EC2 Instance ECS Task Definition ECS Service Definition ECS Cluster 65. ECS Scheduler: Deploys Tasks across the ECS Cluster { "name": "example", "image": "foo/example", "cpu": 1024, "memory": 2048, "essential": true, } { "cluster": "example", "serviceName": âfoo", "taskDefinition": "", "desiredCount": 2 } ECS Agent ECS Tasks EC2 Instance ECS Task Definition ECS Service Definition ECS Scheduler ECS Cluster 66. You can associate an ALB or ELB with each ECS service { "name": "example", "image": "foo/example", "cpu": 1024, "memory": 2048, "essential": true, } { "cluster": "example", "serviceName": âfoo", "taskDefinition": "", "desiredCount": 2 } ECS Agent ECS Tasks EC2 Instance ECS Task Definition ECS Service Definition ECS Cluster 67. This allows you to distribute load across your ECS Tasks { "name": "example", "image": "foo/example", "cpu": 1024, "memory": 2048, "essential": true, } { "cluster": "example", "serviceName": âfoo", "taskDefinition": "", "desiredCount": 2 } ECS Agent ECS Tasks EC2 Instance ECS Task Definition ECS Service Definition ECS Cluster 68. You can also use it as a simple form of service discovery { "name": "example", "image": "foo/example", "cpu": 1024, "memory": 2048, "essential": true, } { "cluster": "example", "serviceName": âfoo", "taskDefinition": "", "desiredCount": 2 } ECS Agent ECS Tasks EC2 Instance ECS Task Definition ECS Service Definition ECS Cluster 69. Letâs deploy our apps on ECS using Terraform 70. Define the ECS Cluster as an Auto Scaling Group (ASG) EC2 Instance ECS Cluster 71. resource "aws_ecs_cluster" "example_cluster" { name = "example-cluster" } resource "aws_autoscaling_group" "ecs_cluster_instances" { name = "ecs-cluster-instances" min_size = 5 max_size = 5 launch_configuration = "${aws_launch_configuration.ecs_instance.name}" } 72. Ensure each server in the ASG runs the ECS Agent ECS Agent EC2 Instance ECS Cluster 73. # The launch config defines what runs on each EC2 instance resource "aws_launch_configuration" "ecs_instance" { name_prefix = "ecs-instance-" instance_type = "t2.micro" # This is an Amazon ECS AMI, which has an ECS Agent # installed that lets it talk to the ECS cluster image_id = "ami-a98cb2c3â } The launch config runs AWS ECS Linux on each server in the ASG 74. Define an ECS Task for each microservice { "name": "example", "image": "foo/example", "cpu": 1024, "memory": 2048, "essential": true, } ECS Agent EC2 Instance ECS Task Definition ECS Cluster 75. resource "aws_ecs_task_definition" "rails_frontend" { family = "rails-frontend" container_definitions = https://www.friendbookmark.com/videos/972/docker-from-zero-to-heroIl container à una VM? Qual'à l'anatomia di un container. Come realizzare il build d'immagine singola e in multi-stage. Cos'à un docker stack e la descrizione d'infrastruttura. In che modo kubernetes realizza un'infrastruttura a microservizi. E chiuderà con lo schema funzionale di un deploy su Docker Desktop con Kubernetes attivo. I temi trattati sono presentati con la citazione di una madrina scelta tra le figure femminili pià rappresentative della storia dell'informatica. Topics covered: 1. DOCKER FROM ZERO TO HERO Sessione Infrastrutturale 2. About Me â Ho esperienza quasi trentennale di sistemi e software in ambito scientifico e della pubblica amministrazione; â Dal 2000 gestisco sistemi in ottica Remoting; â Da 5 anni studio la cultura DevOps ed ho appreso che ogni problema ha soluzione ma la tecnologia non risolve tutti i problemi.latini.giuliano@gmail.com @giulianolatini giulianolatini 3. Agenda Ilcontainer à una VM? Hypervisor Qual'à l'anatomia di un container. Skull&Bones Comerealizzareil build d'immagine singola e in multi-stage Building Cos'à un dockerstack e la descrizione d'infrastruttura Assembly In che modo kubernetes realizza un'infrastruttura a microservizi Orchestration 4. Hypervisor Non à difficile diventare una grande ammaliatrice: basta restare immobile e recitare la parte dellâoca. â âHedy Lamarr inventrice del Frequency Hopping Spread Spectrum, concetto alla base del CDMA e stella di Hollywood 5. STORIA DELLE VM 6. CONCETTO DI ISOLAMENTO Definiti due domini chiamati: INTERNO, ESTERNO e una superficie di separazione chiamata CONFINE, lâISOLAMENTO à dato dalle condizioni applicate sul CONFINE per controllare lo scambio tra INTERNO ed ESTERNO. 7. STATI E LORO PERSISTENZA Unâoperazione che raggiunge sempre lo stesso stato à detta IDEMPOTENTE (Es. 2+2=4). Unâunità operativa che non conserva stati al proprio interno à detta IDEMPOTENTE perchà lâessere re-istanziata dopo un down o uno scaling negativo non causa perdita di stati. 8. STATI E LORO PERSISTENZA 9. CONCETTO DI CONTAINER Un container offre una struttura standardizzata e rilocabile che assolve i prerequisiti del software contenuto. 10. HYPERVISOR A LIVELLO DI SISTEMA OPERATIVO 11. HYPERVISOR A LIVELLO DI SISTEMA OPERATIVO 12. CONTAINER SECONDO L'ASTRAZIONE DOCKER 13. CLI E VERBI DOCKER 14. CLI E VERBI DOCKER 15. ESEMPI D'USO 16. Skull&Bones Trovammo una falena reale[â] la mettemmo nel nostro logbook bloccandola con dello scotch [â] quando qualcosa non andava dicevamo che câerano dei bug e che stavamo âfacendo il debuggingâ â âGrace Hopper prima programmatrice ad introdurre in informatica: il verbo debug e i concetti alla base della frammentazione e riutilizzo del codice; lavorà su Mark I-II-II e UNIVAC fino a gestire il team che produsse COBOL. 17. RUOLO E CICLO DI VITA DI CONTAINER E IMMAGINE 18. CANALI DI COMUNICAZIONE 19. GESTIONE DELLO STORAGE La persistenza in container tramite la creazione di volumi à preferibile alla persistenza su Host perchà i volume plugins permettono lâuso di SAN o filesystem condivisi. 20. GESTIONE DELLO STORAGE 21. TECNOLOGIA COW 22. TECNOLOGIA COW 23. CONCETTI E GESTIONE DEL REGISTRY 24. CONCETTI E GESTIONE DEL REGISTRY 25. Building Desidero spiegare che, con la parola âoperazioneâ, intendiamo qualsiasi processo che modifichi la relazione reciproca di due o pià cose. Questa à la definizione pià generale e includerebbe tutti I soggetti dellâUniverso â âAda Byron contessa di Lovelance programmatrice della macchina analitica di Babbage e geek girl vittoriana 26. VERBO BUILD 27. DOCKERFILE E SUE BEST PRACTICES Lâesecuzione di ogni verbo presente nel Dockerfile produce uno strato nellâimmagine risultante dalla build. 28. DOCKERFILE E SUE BEST PRACTICES 29. DOCKERFILE E SUE BEST PRACTICES 30. BUILD MULTI-STAGE, L'ESEMPIO .NETCORE 31. BUILD MULTI-ARCH, PROSPETTIVE 32. BUILD MULTI-ARCH, PROSPETTIVE 33. BUILD MULTI-ARCH, PROSPETTIVE 34. BUILD MULTI-ARCH, PROSPETTIVE 35. BUILD MULTI-ARCH, PROSPETTIVE 36. Assembly Stiamo avendo unâesplosione di informazioni ed à certamente ovvio che lâinformazione non serve a nulla se non la si rende disponibile a tutti â âSuor Mary Kenneth Keller Educatrice, avvia un dipartimento e un master per le applicazioni dellâinformatica allâinsegnamento, riceve il PhD in informatica nel 1965 con una tesi sulla costruzione di algoritmi in FORTRAN, componente del team di sviluppo del BASIC. 37. CONTAINER SINGOLO E SERVIZIO 38. NETWORKING E STORAGE D'INFRASTRUTTURA 39. NETWORKING E STORAGE D'INFRASTRUTTURA 40. FULLSTACK DESCRITTO NEL DOCKER-COMPOSE.YML 41. INFRASTRUTTURA IN SWARM CON DOCKER STACK $ docker stack deploy --orchestrator kubernetes --compose-file docker-compose.yml vapor-swift-stack 42. Orchestration Penso che le donne portino una prospettiva diversa allâinformatica; sono pià riflessive e meno incline ad andare dritte a soluzioni prettamente tecniche. La mia convinzione à che lâinformatica, intellettualmente, sia molto affascinante sopratutto se hai intenzione di creare qualcosa che non esiste. â âKaren SpÃrck Jones filosofa, ricercatrice in linguistica computazionale e information retrieval. Il concetto di Inverse Document Frequency da lei enunciato nel 1972 à alla base dei motori di ricerca. 43. STORIA E INFRASTRUTTURA DI KUBERNETES 44. STORIA E INFRASTRUTTURA DI KUBERNETES 45. KUBERNETES CHIAVI IN MANO 46. GERARCHIA ELEMENTI IN KUBERNETES 47. INFRASTRUTTURA E SCALING IN KUBERNETES 48. Demo Se le donne vogliono assicurarsi un posto significativo in futuro, devono essere tra quelle che determinano come verrà utilizzata la tecnologia. Devono essere tra quelli che decidono se sarà il grande livellatore o semplicemente serviranno a peggiorare le divisioni sociali. â âAnita Borg PhD in informatica nel 1981; nel 1987, amareggiata dalla scarsa presenza femminile nei convegni e simposi tecnici, fonda la community SYSTERS (SYStem sisTERS) per le donne che lavorano nellâinformatica. Ad oggi conta 7500 membri su 65 paesi . 49. Demo 50. Bibliografia 1/2 â Intro Guide to Dockerfile Best Practices â Docker : A Quick-Start Beginner's Guide (English Edition) â Docker: 5 Books in 1- Beginner's guide+ Tips & Tricks+ Simple & Effective strategies+ Best Practices & Advanced strategies (English Edition) â The Kubernetes Book: Updated Nov 2019 (English Edition) â Docker Deep Dive (English Edition) â Learn Docker - Fundamentals of Docker 18.x 51. Bibliografia 2/2 â Deployment with Docker â Docker in Action, Second Edition â Kubernetes in Action â Microservices with Docker on Microsoft Azure (English Edition) â Scalable Container Infrastructures with Docker, Kubernetes and OpenShift - 2019 Edition (English Edition) 52. Fonte e indice delle citazioni â Hedy Lamarr â Grace Hopper â Ada Byron contessa di Lovelance â Suor Mary Kenneth Keller â Karen SpÃrck Jones â Anita Borg pag. 73 pag. 40 pag. 26 pag. 110 pag. 112 pag. 111 53. Thank you latini.giuliano@gmail.com @giulianolatini giulianolatini https://www.friendbookmark.com/videos/971/from-development-environments-to-production-deployments-with-docker-compose-machine-swarm-and-ecs-cliIn this session, we will learn how to define and run multi-container applications with Docker Compose. Then, we will show how to deploy and scale them seamlessly to a cluster with Docker Swarm; and how Amazon EC2 Container Service (ECS) eliminates the need to install,operate, and scale your own cluster management infrastructure. We will also walk through some best practice patterns used by customers for running their microservices platforms or batch jobs. Sample code and Compose templates will be provided on GitHub afterwards. Topics covered in this presentation slides: 1.  2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. JÃrÃme Petazzoni Docker Inc. From Local Docker Development to Production Deployments 2. What to Expect from the Session We will talk about ... ● Docker Compose for development environments ● taking those environments to production â Docker cluster provisioning â container image building and deployment â service discovery ● Compose, Machine, Swarm, ECS We expect that you are familiar with Docker fundamentals! 3. Introductions ● JÃrÃme Petazzoni (@jpetazzo) ● Since 2010: putting things in containers at dotCloud â polyglot PAAS â microservices â provisioning, metrics, scaling ... â massive deployment of LXC ● Since 2013: putting things in containers at Docker (reminder: dotCloud became Docker in 2013...) ● 5 years of experience on a 2 years old technology! 4. Introductions, take 2 ● Hi, I'm JÃrÃme ● I'm a Software Engineer about to start a new gig! ● Tomorrow for my first day I will work on DockerCoins* ● (It's a cryptocurrency-blockchain-something system) ● My coworkers are using Docker all over the place ● My task will be to deploy their stack at scale *Fictious project name; you can't buy pizzas or coffee with DockerCoins (yet). 5. Getting ready 6. Preparing for my first day ● I just received my new laptop! ● The only instructions where: âInstall the Docker Toolbox.â ● ~180 MB download for Windows and OS X 7. Video: https://www.youtube.com/watch?v=g-g94H_AiOE 8. Developing with Compose 9. The Compose on-boarding workflow ● Three simple steps: 1) git clone 2) docker-compose up 3) open app in browser 10. DEMO Video: https://www.youtube.com/watch?v=sk3yYh1MgE0 11. How does this work? ● âdocker-compose upâ tells Compose to start the app ● If needed, the app is built first ● How does Compose know what to do? ● It reads the âCompose fileâ (docker-compose.yml) 12. docker-compose.yml â simple application web: build: . ports: - "80:5000" links: - redis redis: image: redis 13. docker-compose.yml â complex application rng: build: rng ports: - "8001:80" hasher: build: hasher ports: - "8002:80" redis: image: redis worker: build: worker links: - rng - hasher - redis webui: build: webui links: - redis ports: - "8000:80" volumes: - "./webui/files/:/files/" 14. How does this work? ● Application is broken down into services ● Each service is mapped to a container ● Each container can come from: â a pre-built image in a library called âregistryâ â a build recipe called âDockerfileâ ● The Compose file defines all those services (and their parameters: storage, network, env vars...) 15. DEMO ● show docker-compose.yml file and highlight services ● show a couple of services 16. Our sample application ● Microservices architecture ● Different languages and frameworks â Ruby + Sinatra â Python + Flask â Node.js + Express ● Different kinds of services â background workers â web services with REST API â stateful data stores â web front-ends 17. Mandatory plug on microservices ● Advantages of microservices: â enables small teams (Jeff Bezos two-pizza rule) â enables âright tool for the right jobâ â services can be deployed/scaled independently â look for e.g. âAdrian Cockroft Microservicesâ talks ● Drawbacks to microservices: â distributed systems are hard (cf. aphyr.com if you have doubts) â load balancing, service discovery become essential â look for e.g. âMicroservices Not A Free Lunchâ article 18. Deploying on a Cloud Instance ● Same workflow: 1) ssh into remote Docker Host 2) git clone 3) docker-compose up 4) open app in browser ● Let's see a real demo! 19. DEMO ● ssh and repeat 20. DEMO ● git clone git://github.com/jpetazzo/dockercoins ● cd dockercoins ● docker-compose up ● open app (instance address, port 8000) ● ^C 21. The Compose development workflow ● Four simple steps: 1) edit code 2) docker-compose build 3) docker-compose up 4) reload app in browser 22. DEMO ● edit webui/files/index.html ● change css ● docker-compose build ● docker-compose up ● reload app ● ^C Video: https://www.youtube.com/watch?v=O3Bps01THBQ 23. Compose take-aways ● Docker abstracts the environment for us ● Any Docker host is a valid deployment target: â local environment (with the Docker Toolbox) â on-demand cloud instances (with Docker Machine) â bring-Your-Own-Server (for on-prem and hybrid strategies) ● Frictionless on-boarding (and context-switching) ● But how do we deploy to production, at scale? 24. What's missing ● Cluster provisioning ● Building and deploying code ● Service discovery (Non-exhaustive list.) Let's see how to address those points. We will dive into details â and give more live demos! 25. Provisioning a cluster 26. Provisioning ● Manual instance creation (CLI or Console) ● AWS CLI scripting ● Auto Scaling Groups ● CloudFormation templates ● Docker Machine ● ECS CLI 27. Docker Machine ● Docker Machine comes with the Docker Toolbox ● Can create Docker hosts on: â EC2 and other clouds â local environments (VirtualBox, OpenStackâ) ● Can create clusters using Docker Swarm ● Current limitations (but expect this to improve): â one machine at a time â centralized credentials 28. DEMO export TOKEN=$(docker run swarm create) echo $TOKEN docker-machine create -d amazonec2 --swarm --swarm-master --swarm-discovery token://$TOKEN node00 & for N in $(seq 1 4); do sleep 3 docker-machine create -d amazonec2 --swarm --swarm-discovery token://$TOKEN node0$N & done wait Video: https://www.youtube.com/watch?v=LFjwusorazs 29. ECS CLI ● Sneak peek! ● State-of-the-art cluster creation ● Following AWS best practices: â CloudFormation template â Auto Scaling Group â IAM integration 30. DEMO ● ecs-cli configure ● ecs-cli up --keypair jpetazzo --capability-iam --size 10 ● (add elastic load balancer) ● (associate load balancer with auto scaling group) ● (add DNS entry) ● (configure security groups for ELB and ASG) Video: https://www.youtube.com/watch?v=KqEpIDFxjNc 31. Building and deploying code 32. Building and deploying with Docker ● Let's continue to use Compose to build our app images ● And store those images in a Docker Registry â Docker Hub (SAAS à la GitHub, free for public images) â Docker Trusted Registry (commercial offering; available e.g. through AWS marketplace) â self-hosted, community version 33. The plan ● Each time we need to deploy: 1) build all containers with Compose 2) tag all images with a unique version number 3) push all images to our Registry 4) generate a new docker-compose.yml file, referencing the images that we just built and pushed ● This will be done by a script 34. You get a script! And you get a script! Everybody gets a script! ● All the scripts that we will use here are on GitHub ● Feel free to use them, copy them, adapt them, etc. URL: https://github.com/jpetazzo/orchestration-workshop (Don't panic, URL will be shown again at the end of the presentation) 35. DEMO ● build-tag-push.py ● inspect the resulting YAML file Those images are now frozen. They'll stay around âforeverâ if we need them again. (e.g. to do a version rollback) See: https://hub.docker.com/r/jpetazzo/dockercoins_webui/tags/ 36. Service discovery 37. Why do we need service discovery? ● Service A needs to talk to service B ● How does A know how to talk to B? â service A needs: address, port, credentials ● What if there are multiple instances of B? â examples: load balancing, replication ● What if B location changes over time? â examples: scaling, fail-over ● Service discovery addresses those concerns 38. Service discovery, seen by devs 39. Hard-coded service discovery ● Development setup: $db = mysql_connect(âlocalhostâ); cache = Redis.new(:host => "localhost", :port => 16379) conn, err := net.Dial("tcp", "localhost:8000â) 40. Hard-coded service discovery ● Other development setup: $db = mysql_connect(â192.168.1.2â); cache = Redis.new(:host => "192.168.1.3", :port => 6380) conn, err := net.Dial("tcp", "192.168.1.4:8080â) 41. Hard-coded service discovery ● Production setup: $db = mysql_connect( âfoo.rds.amazonaws.comâ, âproduserâ, âsesameâ); cache = Redis.new( :url => "redis://:p4ssw0rd@redis-as-a-service.io/15â) conn, err := net.Dial( "tcp", "api-42.elb.amazonaws.com:80â) 42. Hard-coded service discovery ● Requires many code edits to change environment ● Error-prone ● Big, repetitive configuration files often land in the repo ● Adding a new service requires editing all those configs ● Maintenance is expensive (S services à E environments) ● � 43. Twelve-factor App ● Environment variables $db = mysql_connect( $_ENV[âDB_HOSTâ], $_ENV[âDB_USERâ], $_ENV[âDB_PASSâ]) cache = Redis.new( :url => "redis://:#{ENV[âREDIS_PASSâ]}@â + â#{ENV[âREDIS_HOST]}:#{ENV[âREDIS_PORT]}/â + â#{ENV[âREDIS_DBâ]}â) conn, err := net.Dial( "tcp", os.ExpandEnv("${API_HOST}:${API_PORT}â)) 44. Twelve-factor App ● Separates cleanly code and environment variables (environment is literally defined by environment variables) ● Still requires to maintain configuration files (containing lists of environment variables) ● Production parameters are easier to keep out of the repo ● Dramatic errors are less likely to happen ● � 45. Configuration database ● Dynamic lookups (here with Zookeeper) $zk = new Zookeeper('127.0.0.1:2181'); mysql_connect( $zkâget('/apps/foo/prod/db/host') $zkâget('/apps/foo/prod/db/user') $zkâget('/apps/foo/prod/db/pass')) zk = Zookeeper.new('127.0.0.1:2181') redis_pass = zk.get(:path => '/apps/foo/prod/redis/pass') redis_host = zk.get(:path => '/apps/foo/prod/redis/host') redis_port = zk.get(:path => '/apps/foo/prod/redis/port') redis_db = zk.get(:path => '/apps/foo/prod/redis/db') cache = Redis.new( :url => "redis://:#{redis_pass}@#{redis_host}:#{redis_port}/#{redis_db}â) c, _, err := zk.Connect([]string{"127.0.0.1"}, time.Second) api_host, _, err := c.get(â/apps/foo/prod/api/hostâ) api_port, _, err := c.get(â/apps/foo/prod/api/portâ) conn, err := net.Dial(âtcpâ', fmt.Sprintf(â%s:%sâ, api_host, api_port)) 46. Configuration database ● If you want the same code in dev and prod, you need to deploy your config DB in dev too ● Instead of maintaining config files, you maintain Zookeeper* clusters and fixtures ● â or have different lookup logic for dev and prod ● � *Or your other favorite config DB, e.g. etcd, Consul... 47. Local load balancing / routing ● Connect to well-known location $db = mysql_connect(âlocalhostâ); cache = Redis.new(:host => "localhost") conn, err := net.Dial("tcp", "localhost:8001â) ● In dev: all components run locally ● In prod: local load balancer routes the traffic (example: AirBNB's SmartStack) 48. Local load balancing / routing ● Code can be identical in dev and prod ● Deployment will differ: â direct connection in dev â proxies, routers, load balancers in prod ● âConfigurationâ is merely a static port allocation map (indicating which service listens on which port) ● Way easier for devs; however ops still have work to do ● � 49. The ambassador pattern 50. Our code base with ambassadors ● Use well-known DNS names $db = mysql_connect(âdbâ); cache = Redis.new(:host => "redis") conn, err := net.Dial("tcp", "api:80â) 51. Running in dev worker 172.17.0.4 worker 172.17.0.4 rng 172.17.0.2 rng 172.17.0.2 hasher 172.17.0.1 hasher 172.17.0.1 webui 172.17.0.5 webui 172.17.0.5 redis 172.17.0.3 redis 172.17.0.3 Let's populate a custom /etc/hosts file in each container, referencing the services that it needs to connect to. e.g. on âworkerâ: 172.17.0.1 hasher 172.17.0.2 rng 172.17.0.3 redis ContainerContainerHostHost 52. Another dev environment worker 10.0.0.20 worker 10.0.0.20 rng 10.0.0.15 rng 10.0.0.15 hasher 10.0.0.10 hasher 10.0.0.10 webui 10.0.0.4 webui 10.0.0.4 redis 10.0.0.12 redis 10.0.0.12 The addressing is different, but the code remains the same. /etc/hosts on âworkerâ: 10.0.0.10 hasher 10.0.0.15 rng 10.0.0.12 redis ContainerContainerHostHost 53. Some good news worker 172.17.0.4 worker 172.17.0.4 rng 172.17.0.2 rng 172.17.0.2 hasher 172.17.0.1 hasher 172.17.0.1 webui 172.17.0.5 webui 172.17.0.5 redis 172.17.0.3 redis 172.17.0.3 Compose automatically does this for us, using Docker âlinks.â Links populate /etc/hosts. Our dev environment is already taken care of! But what about our production setup on multiple hosts? ContainerContainerHostHost 54. Running in prod worker 172.17.0.4 worker 172.17.0.4 rng 172.17.0.2 rng 172.17.0.2 hasher 172.17.0.1 hasher 172.17.0.1 redis 172.17.0.3 redis 172.17.0.3 Worker doesn't talk to actual instances of redis, hasher, and rng, but to ambassadors. Ambassadors will route* the traffic to the destination. *Or forward, load-balance, proxy... ContainerContainerHostHost AmbassadorAmbassador 55. Running in prod worker 172.17.0.4 worker 172.17.0.4 rng 172.17.0.2 rng 172.17.0.2 hasher 172.17.0.1 hasher 172.17.0.1 redis 172.17.0.3 redis 172.17.0.3 redis 172.17.0.6 redis 172.17.0.6 hasher 172.17.0.8 hasher 172.17.0.8 rng 172.17.0.5 rng 172.17.0.5 rng 172.17.0.4 rng 172.17.0.4 webui 172.17.0.8 webui 172.17.0.8 redis 172.17.0.7 redis 172.17.0.7 ContainerContainerHostHost AmbassadorAmbassador 56. Using ambassadors ● Code remains readable and clean ● Plumbing (service discovery, routing, load balancing, etc.) is abstracted away (somebody still has to do it, though!) ● Plumbing doesn't encumber our dev environment ● Changes in plumbing won't impact the code base ● � 57. Service discovery, seen by ops 58. How fast are we moving? 59. Moving slowly ● Code deployment is infrequent: â every week, on a regular schedule â a bit of downtime is OK (a few minutes, maybe one hour) ● Failures are rare (less than 1/year) and/or don't have critical impact ● Reconfigurations are not urgent: â we bake them in the deployment process â it's OK if they disrupt service or cause downtime 60. Strategy for apps moving slowly ● Bake configuration and parameters with the deployment (reconfiguration = rebuild, repush, redeploy) ● Or configure manually after deployment (!) ● In case of emergency: SSH+vi (!) 61. Results ● Advantages â zero cost upfront â easy to understand* ● Drawbacks â each deployment, each change = risk â expensive in the long run *Except for your boss when your app is down and it takes a while to bring it back up 62. Moving mildly ● Code deployment: â happens every day â downtime is not OK (except maybe very short glitches) ● Failures happen regularly; they must be resolved quickly ● Reconfigurations are frequent: â scaling up/down; moving workloads; changing databases â altering application parameters for A/B testing 63. Strategy for apps moving mildly ● Inject configuration after the deployment ● When you just want to change a parameter: reconfigure (without redeploying everything) ● Automate the process with a âpush buttonâ script 64. Results ● Advantages â easy to understand and to implement â no extra moving part (just this extra âpush buttonâ script/process) ● Drawbacks â services must allow reconfiguration â reconfiguration has to be triggered after each change â risk of meta-failure (bug in the deployment system) 65. Moving wildly* ● Code deployment: â happens continuously (10, 100, 1000+ times a day) â downtime is not OK, even if it's just a few sporadic failed requests ● Failures happen all the time; repair actions must be fully automated ● Reconfigurations are part of the app lifecycle: â automatic scaling, following planned and unplanned patterns â generalized blue/green deployment, canary testing, etc. *a.k.a âmove fast and break thingsâ 66. Strategy for apps moving wildly ● Requirement: detect changes as they happen ● Use a combination of: â monitoring â live stream of events that we can subscribe to â services that register themselves â fast polling ● After deployment, scaling, outage, metric thresholdâ: automatic reconfiguration 67. Results ● Advantages â everything happens automatically â no extra step to run when you deploy â more modular (different processes can take care of different service types) ● Drawbacks â extra moving parts and services to maintain â meta-failures are even more dangerous 68. Recap table How fast should we move? How much work is it for ... How do we handle ... Slowly Mildly Wildly Devs Ops Scaling Failures Hard-coded 12-Factor Config Database Local LB/routers Ambassadors 69. Recap table (subtitles) How fast should we move? How much work is it for ... How do we handle ... Slowly Mildly Wildly Devs Ops Scaling Failures Hard-coded OK NO NO easy easy painfully horribly 12-Factor OK OK WITH RESTARTS NO easy easy meh meh Config Database OK OK OK hard hard cool cool Local LB/routers OK OK OK medium medium /hard cool cool Ambassadors OK OK OK easy medium /hard cool cool 70. Ambassadors in action 71. The plan ● Deploy a simple application (trainingwheels) â on ECS â on Swarm ● Deploy a complex application (dockercoins) â on ECS â on Swarm 72. Our simple application, âtrainingwheelsâ ● Two service: â web server â redis data store ● Tells you which web server served your request ● Counts how many requests were served ● Keeps separate counters for each server 73. DEMO ● cd ~ ● git clone git://github.com/jpetazzo/trainingwheels ● cd trainingwheels ● docker-compose up ● open app ● ^C 74. Deploying on ECS ● On ECS, a container is created as a member of a task ● Tasks are created from task definitions ● Task definitions are conceptually similar to Compose files (but in a different format) ● ECS CLI to the rescue! 75. Deploying on ECS ● ECS CLI will: â create a task definition from our Compose file â register that task definition with ECS â run a task instance from that task definition ● ECS CLI will not: â work if your Compose file has a âbuildâ section (it only accepts âimageâ sections) ● Let's use the âbuild-tag-pushâ script shown earlier! 76. DEMO ● build-tag-push.py ● set COMPOSE_FILE ● fixup-yaml.sh 77. Scaling âtrainingwheelsâ on ECS At this point, if we deploy and scale, we will end up with multiple copies of the app, each with its own Redis. To avoid this, we need to deploy our first ambassador! Here is the plan: ● Create a new Compose file for our Redis service ● Use ECS CLI to run redis, and note its location ● Update the main Compose file so that the âredisâ service is now an ambassador pointing to the actual Redis 78. Introducting jpetazzo/hamba ● Easy ambassadoring for the masses! ● In a shell: docker run jpetazzo/hamba [backend1-addr] [backend1-port] [backend2-addr] [backend2-port] â ● In a Compose file: redis: image: jpetazzo/hamba command: [backend-addr] [backend-port] ... 79. DEMO (1/2) ● mkdir ~/myredis ● cp $COMPOSE_FILE ~/myredis ● cd ~/myredis ● edit $COMPOSE_FILE â expose port 6379 â remove www service ● ecs-cli compose up ● ecs-cli compose ps ● note host+port 80. DEMO (2/2) ● cd ~/trainingwheels ● edit $COMPOSE_FILE â replace redis image with jpetazzo/hamba â add âcommand: 6379 â ● ecs-cli compose up ● ecs-cli compose scale 4 ● watch ecs-cli compose ps ● open a couple of apps ● open the load balancer 81. CLEANUP ● ecs-cli compose down ● Let redis running (we'll re-use it later) 82. Scaling âtrainingwheelsâ on Swarm ● Slightly different idea! ● We keep a single Compose file for our app ● We replace links with ambassadors: â using a local address (127.X.Y.Z) â sharing the client container's namespace ● Each container that needs to connect to another service, gets its own private load balancer for this exact service ● That's a lot of load balancers, but don't worry, they're cheap 83. Network namespace ambassadors www 172.17.0.4 www 172.17.0.4 âredisâ and âwwwâ containers are created by Compose, and placed by Swarm, potentially on different hosts. In âwwwâ, /etc/hosts has the following entry: 127.127.0.2 redis ContainerContainerHostHost AmbassadorAmbassador redis 172.17.2.5 redis 172.17.2.5 84. Network namespace ambassadors www 172.17.0.4 www 172.17.0.4 At this stage, connection attempts from âwwwâ to âredisâ fail with âconnection refused.â ContainerContainerHostHost AmbassadorAmbassador redis 172.17.2.5 redis 172.17.2.5 85. Network namespace ambassadors www 172.17.0.4 www 172.17.0.4 ambassador 127.127.0.2 ambassador 127.127.0.2 The ambassador is created. It's sharing the network namespace of the âwwwâ container, meaning that they have the same loopback interface (they can talk over localhost). ContainerContainerHostHost AmbassadorAmbassador redis 172.17.2.5 redis 172.17.2.5 86. Network namespace ambassadors www 172.17.0.4 www 172.17.0.4 ambassador 127.127.0.2 ambassador 127.127.0.2 At this stage, connections are still failing (with either âconnection refusedâ or a timeout, depending on the load balancer settings.) The application has to handle this gracefully. (Crashing and being restarted is graceful enough.) ContainerContainerHostHost AmbassadorAmbassador redis 172.17.2.5 redis 172.17.2.5 87. Network namespace ambassadors www 172.17.0.4 www 172.17.0.4 ambassador 127.127.0.2 ambassador 127.127.0.2 The ambassador receives its configuration, containing the public address of the âredisâ container. ContainerContainerHostHost AmbassadorAmbassador redis 172.17.2.5 redis 172.17.2.5 88. Network namespace ambassadors www 172.17.0.4 www 172.17.0.4 ambassador 127.127.0.2 ambassador 127.127.0.2 Traffic can now flow normally from âwwwâ to âredisâ. ContainerContainerHostHost AmbassadorAmbassador redis 172.17.2.5 redis 172.17.2.5 89. DEMO (1/2) ● eval $(docker-machine env node00 --swarm) ● edit $COMPOSE_FILE â revert âredisâ to use âimage: redisâ â remove âcommand:â ● link-to-ambassadors.py ● docker-compose up -d ● docker-compose ps ● open app ● (It doesn't work â yet) 90. DEMO (2/2) ● create-ambassadors.py ● configure-ambassadors.py ● open app ● docker-compose scale www=4 ● create-ambassadors.py ● configure-ambassadors.py ● open app 91. Scaling with ambassadors Before scaling our app, we have one single âwwwâ instance, coupled with its ambassador. (In this example, we have placed the first âwwwâ and âredisâ together for clarity.) ContainerContainerHostHost AmbassadorAmbassador wwwwww redisredis ambaamba 92. Scaling with ambassadors âdocker-compose scale www=4â We now have 4 instances of âwwwâ but 3 of them can't communicate with âredisâ yet. ContainerContainerHostHost AmbassadorAmbassador wwwwww wwwwww wwwwww wwwwww redisredis ambaamba 93. Scaling with ambassadors âcreate-ambassadors.pyâ Each âwwwâ instance now has its own ambassador, but 3 of them are still unconfigured. ContainerContainerHostHost AmbassadorAmbassador wwwwww wwwwww ambaamba wwwwww ambaamba redisredis ambaamba wwwwww ambaamba 94. Scaling with ambassadors âconfigure-ambassadors.pyâ The 3 new ambassadors receive their configuration and can now route traffic to the âredisâ service. ContainerContainerHostHost AmbassadorAmbassador wwwwww wwwwww ambaamba wwwwww ambaamba redisredis ambaamba wwwwww ambaamba 95. CLEANUP ● docker-compose kill ● docker-compose rm -f 96. Scaling âdockercoinsâ on ECS ● Let's apply the same technique as before ● Separate the Redis service ● Replace âredisâ with an ambassador in the Compose file ● Let ECS do the rest! 97. DEMO (1/2) ● Get our Redis host+port again: â cd ~/myredis â ecs-cli compose ps ● cd ~/dockercoins ● set COMPOSE_FILE ● edit $COMPOSE_FILE â change âimage: redisâ to âimage: jpetazzo/hambaâ â add âcommand: 6379 â add âmem_limit: 100000000â everywhere â remove volumes ● fixup-yaml.sh 98. DEMO (2/2) ● ecs-cli compose up ● watch ecs-cli compose ps ● open webui ● ecs-cli compose scale 4 ● watch ecs-cli compose ps ● open webui ● repeat! 99. Scaling âdockercoinsâ on ECS ● We started with our âredisâ service... ContainerContainerHostHost AmbassadorAmbassador redisredis 100. Scaling âdockercoinsâ on ECS ● Created one instance of the stack with an ambassador... ContainerContainerHostHost AmbassadorAmbassador workerworker rngrnghasherhasher webuiwebui redisredis redisredis 101. Scaling âdockercoinsâ on ECS workerworker rngrnghasherhasher webuiwebui ● Added a second instance of the full stack... ContainerContainerHostHost redisredis AmbassadorAmbassador workerworker rngrnghasherhasher webuiwebui redisredis redisredis 102. Scaling âdockercoinsâ on ECS workerworker rngrnghasherhasher webuiwebui ● And another oneâ etc. ContainerContainerHostHost redisredis AmbassadorAmbassador workerworker rngrnghasherhasher webuiwebui redisredis workerworker rngrnghasherhasher webuiwebui redisredis redisredis 103. Scaling âdockercoinsâ on Swarm ● Let's apply the same technique as before ● Replace links with ambassadors ● Start containers ● Add ambassadors ● Inject ambassador configuration 104. DEMO (1/2) ● edit COMPOSE_FILE â restore âimage: redisâ â remove âcommand:â from the redis section ● link-to-ambassadors.py ● docker-compose up -d ● create-ambassadors.py ● configure-ambassadors.py ● docker-compose ps webui ● open webui 105. DEMO (2/2) ● docker-compose scale webui=2 worker=10 rng=20 hasher=5 ● create-ambassadors.py ● configure-ambassadors.py 106. Scaling âdockercoinsâ on Swarm ● Two (for simplicity) empty Docker hosts ContainerContainerHostHost AmbassadorAmbassador 107. Scaling âdockercoinsâ on Swarm ● âdocker-compose upâ â containers are unwired ContainerContainerHostHost AmbassadorAmbassador workerworker webuiwebui redisredis hasherhasher rngrng 108. Scaling âdockercoinsâ on Swarm ● Create ambassadors for all containers needing them ContainerContainerHostHost AmbassadorAmbassador workerworker webuiwebui redisredis redisredis hasherhasher rngrng redisredis hasherhasher rngrng 109. Scaling âdockercoinsâ on Swarm ● Configure ambassadors: the app is up and running ContainerContainerHostHost AmbassadorAmbassador workerworker webuiwebui redisredis redisredis hasherhasher rngrng redisredis hasherhasher rngrng 110. Scaling âdockercoinsâ on Swarm ● âdocker-compose scaleâ ContainerContainerHostHost AmbassadorAmbassador workerworker webuiwebui redisredis redisredis hasherhasher rngrng redisredis workerworker hasherhasher rngrng workerworker hasherhasher rngrng rngrng hasherhasher rngrng rngrng 111. Scaling âdockercoinsâ on Swarm ● Creation of new ambassadors ContainerContainerHostHost AmbassadorAmbassador workerworker webuiwebui redisredis redisredis hasherhasher rngrng redisredis workerworker redisredis hasherhasher rngrng hasherhasher rngrng workerworker redisredis hasherhasher rngrng hasherhasher rngrng rngrng hasherhasher rngrng rngrng 112. Scaling âdockercoinsâ on Swarm ● Configuration of new ambassadors ContainerContainerHostHost AmbassadorAmbassador workerworker webuiwebui redisredis redisredis hasherhasher rngrng redisredis workerworker redisredis hasherhasher rngrng hasherhasher rngrng workerworker redisredis hasherhasher rngrng hasherhasher rngrng rngrng hasherhasher rngrng rngrng 113. Remarks ● Yes, that's a lot of ambassadors ● They are very lightweight, though (~1 MB) docker stats $(docker ps | grep hamba | awk '{print $1}') ● Ambassadors do not add an extra hop â they are local to their client (virtually zero latency) â better efficiency than external load balancer â if the ambassador is down, the client is probably down as well 114. Recap 115. The Docker Compose workflow ● Application portability between: â Docker Compose + Docker Toolbox â Docker Compose + Docker Swarm â ECS CLI + ECS ● Interface points: â Compose file â Docker Registry 116. ECS and Swarm highlights ● Both offer easy provisioning tools ● ECS = AWS ecosystem â integrates with offerings like IAM, ELBâ â provides health-checks and self-healing ● Swarm = Docker ecosystem â offers parity with local development environments â exposes real-time events stream through Docker API ● Both require additional tooling for builds (Swarm has preliminary build support) ● Both require extra work for plumbing / service discovery 117. Future directions, ideas ... ● We would love your feedback! ● App-specific ambassadors (SQL bouncers, credential injectors...) ● Automatically replace services using official images: â redis, memcached â elasticache â mysql, postgresql â RDS â etc. 118. Other improvements ● Listen to Docker API events stream, detect containers start/stop events â automatically configure load balancers (ehazlett/interlock) â insert containers into a config database (gliderlabs/registrator) ● Overlay networks (offers direct container-to-container communication) â 3rd party: weave, flannel, pipework â Docker network plugins (experimental.docker.com) 119. Thank you! Questions? 120. Remember to complete your evaluations! 121. Thank you! Related sessions ● CMP302 - Amazon EC2 Container Service: Distributed Applications at Scale ● CMP406 - Amazon ECS at Coursera: Powering a general-purpose near-line execution microservice, while defending against untrusted code ● DVO305 - Turbocharge Your Continuous Deployment Pipeline with Containers ● DVO308 - Docker & ECS in Production: How We Migrated Our Infrastructure from Heroku to AWS (Remind) ● DVO313 - Building Next-Generation Applications with Amazon ECS (Meteor) 122. Thank you! Code repositories: https://github.com/aws/amazon-ecs-cli https://github.com/jpetazzo/dockercoins https://github.com/jpetazzo/trainingwheels https://github.com/jpetazzo/orchestration-workshop Videos: https://www.youtube.com/watch?v=g-g94H_AiOE https://www.youtube.com/watch?v=sk3yYh1MgE0 https://www.youtube.com/watch?v=O3Bps01THBQ https://www.youtube.com/watch?v=LFjwusorazs https://www.youtube.com/watch?v=KqEpIDFxjNc Note: videos just include the installation and deployment processes. I'll make videos of the other demos if there's enough demand for it! Follow us on Twitter: @docker @jpetazzo https://www.friendbookmark.com/videos/970/package-your-java-ee-application-using-docker-and-kubernetesPackage your Java EE Application using Docker and Kubernetes. Topics covered in this presentation slides: 1. Package your
 Java EE applications
 using
 Docker and Kubernetes
 Arun Gupta, @arungupta Red Hat 2. Arun Gupta Director, Developer Advocacy @arungupta blog.arungupta.me arungupta@redhat.com 3. What is Docker? 4. What is Docker? â Open source project and company
 
 
 
 
 
 5. What is Docker? â Open source project and company
 
 
 
 
 
 â Used to create containers for software applications 6. What is Docker? â Open source project and company
 
 
 
 
 
 â Used to create containers for software applications â Package Once Deploy Anywhere (PODA) 7. Advantages 8. Advantages â Faster deployments 9. Advantages â Faster deployments â Isolation 10. Advantages â Faster deployments â Isolation â Portability - âit works on my machineâ 11. Advantages â Faster deployments â Isolation â Portability - âit works on my machineâ â Snapshotting 12. Advantages â Faster deployments â Isolation â Portability - âit works on my machineâ â Snapshotting â Security sandbox 13. Advantages â Faster deployments â Isolation â Portability - âit works on my machineâ â Snapshotting â Security sandbox â Limit resource usage 14. Advantages â Faster deployments â Isolation â Portability - âit works on my machineâ â Snapshotting â Security sandbox â Limit resource usage â Simplified dependency 15. Advantages â Faster deployments â Isolation â Portability - âit works on my machineâ â Snapshotting â Security sandbox â Limit resource usage â Simplified dependency â Sharing 16. Underlying Technology 17. Underlying Technology â Written in Go
 18. Underlying Technology â Written in Go
 â Uses several Linux features 19. Underlying Technology â Written in Go
 â Uses several Linux features â Namespaces to provide isolation 20. Underlying Technology â Written in Go
 â Uses several Linux features â Namespaces to provide isolation â Control groups to share/limit hardware resources 21. Underlying Technology â Written in Go
 â Uses several Linux features â Namespaces to provide isolation â Control groups to share/limit hardware resources â Union File System makes it light and fast 22. Underlying Technology â Written in Go
 â Uses several Linux features â Namespaces to provide isolation â Control groups to share/limit hardware resources â Union File System makes it light and fast â libcontainer defines container format 23. Is it only Linux? 24. Is it only Linux? â Natively supported in Linux 25. Is it only Linux? â Natively supported in Linux â Can be installed on Mac or Windows using boot2docker 26. Is it only Linux? â Natively supported in Linux â Can be installed on Mac or Windows using boot2docker â Tiny Core Linux VM 27. â Image defined in text-based Dockerfile 28. â Image defined in text-based Dockerfile â List of commands to build the image
 
 
 
 FROM fedora:latest
 
 CMD echo âHello worldâ 29. â Image defined in text-based Dockerfile â List of commands to build the image
 
 
 
 â docker build or pull FROM fedora:latest
 
 CMD echo âHello worldâ 30. â Images shared using registry 31. â Images shared using registry â Docker Hub is public SaaS
 
 
 
 
 
 
 32. â Images shared using registry â Docker Hub is public SaaS
 
 
 
 
 
 
 â Private registries can be setup inside firewall 33. â Images shared using registry â Docker Hub is public SaaS
 
 
 
 
 
 
 â Private registries can be setup inside firewall â docker push or pull 34. â Container built from the image
 
 
 
 
 
 
 35. â Container built from the image
 
 
 
 
 
 
 â Runtime representation of the image 36. â Container built from the image
 
 
 
 
 
 
 â Runtime representation of the image â Self contained execution environment 37. â Container built from the image
 
 
 
 
 
 
 â Runtime representation of the image â Self contained execution environment â docker run 38. Docker commands â docker ps: List running containers â docker stop: Stop a running container â docker rm: Remove a running container â docker rmi: Remove an image â â https://docs.docker.com/reference/commandline/cli/ 39. Docker
 Hub
 Docker
 Host
 DaemonDocker
 Client Docker Workflow 40. Docker
 Hub
 Docker
 Host
 DaemonDocker
 Client docker run docker â Docker Workflow 41. Docker
 Hub
 Docker
 Host
 DaemonDocker
 Client docker run docker â Docker Workflow 42. Docker
 Hub
 Image 1 Image 2 Image 3 Image M Docker
 Host
 DaemonDocker
 Client docker run docker â Docker Workflow 43. Docker
 Hub
 Image 1 Image 2 Image 3 Image M Docker
 Host
 Image 1 Image 2 Image 3 Image N DaemonDocker
 Client docker run docker â Docker Workflow 44. Docker
 Hub
 Image 1 Image 2 Image 3 Image M Docker
 Host
 Image 1 Image 2 Image 3 Image N Daemon Container 1 Container 2 Container O Docker
 Client docker run docker â Docker Workflow 45. Recipe #1.1 FROM jboss/wildfly RUN curl -L https://github.com/javaee-samples/javaee7-hol/raw/master/solution/ movieplex7-1.0-SNAPSHOT.war -o /opt/jboss/wildfly/standalone/deployments/ movieplex7-1.0-SNAPSHOT.war docker run -it -p 8080:8080 arungupta/javaee7-hol Host
 Application Server Database 46. Recipe #1.2 Host
 Application Server Database http://blog.arungupta.me/wildfly-javaee7-mysql-link-two-docker-container-techtip65/ data-source add --name=mysqlDS --driver-name=mysql --jndi-name=java:jboss/ datasources/ExampleMySQLDS --connection-url=jdbc:mysql://$DB_PORT_3306_TCP_ADDR: $DB_PORT_3306_TCP_PORT/sample?useUnicode=true&characterEncoding=UTF-8 -- user-name=mysql --password=mysql --use-ccm=false --max-pool-size=25 --blocking- timeout-wait-millis=5000 --enabled=true 47. Recipe #1.3 Host
 Application Server Database http://blog.arungupta.me/docker-orchestration-fig-techtip67/ 48. Recipe #1.4 Host
 Application Server http://blog.arungupta.me/docker-container-linking-across-multiple-hosts-techtip69/ Host
 Database 49. Recipe #1.4 Host
 Application Server http://blog.arungupta.me/docker-container-linking-across-multiple-hosts-techtip69/ Host
 Database 50. Recipe #1.4 Host
 Application Server http://blog.arungupta.me/docker-container-linking-across-multiple-hosts-techtip69/ Host
 Database 51. Recipe #1.4 Host
 Application Server http://blog.arungupta.me/docker-container-linking-across-multiple-hosts-techtip69/ Host
 Database 52. Arquillian Cube â Controls the lifecycle of Docker images as part of test cycle - automatically or manually â Uses Docker REST API to talk to container â Talk using WildFly remote adapter (in container) â Try it out http://blog.arungupta.me/run-javaee-tests-wildfly-docker-arquillian-cube/ 53. Docker: Pros and Cons 54. Docker: Pros and Cons â PROS â Extreme application portability â Very easy to create and work with derivative â Fast boot on containers 55. Docker: Pros and Cons â PROS â Extreme application portability â Very easy to create and work with derivative â Fast boot on containers â CONS â Host-centric solution â No higher-level provisioning â No usage tracking/reporting 56. Application Operating Environment 57. Kubernetes 58. Kubernetes â Open source orchestration system for Docker containers 59. Kubernetes â Open source orchestration system for Docker containers â Provide declarative primitives for the âdesired stateâ â Self-healing â Auto-restarting â Schedule across hosts â Replicating 60. Concepts 61. Concepts â Pods: collocated group of Docker containers that share an IP and storage volume Docker Pod 1 Pod 2 C1 C2 C3 62. Concepts â Pods: collocated group of Docker containers that share an IP and storage volume â Service: Single, stable name for a set of pods, also acts as LB Docker Pod 1 Pod 2 C1 C2 C3 Pod 1 JBoss Pod 2 JBoss Service âwebâ port 8080 port 8080 63. Concepts â Pods: collocated group of Docker containers that share an IP and storage volume â Service: Single, stable name for a set of pods, also acts as LB â Replication Controller: manages the lifecycle of pods and ensures specified number are running Docker Pod 1 Pod 2 C1 C2 C3 Pod 1 JBoss Pod 2 JBoss Service âwebâ port 8080 port 8080 64. Concepts â Pods: collocated group of Docker containers that share an IP and storage volume â Service: Single, stable name for a set of pods, also acts as LB â Replication Controller: manages the lifecycle of pods and ensures specified number are running â Label: used to organize and select group of objects Docker Pod 1 Pod 2 C1 C2 C3 Pod 1 JBoss Pod 2 JBoss Service âwebâ port 8080 port 8080 65. kubectl 66. kubectl â Controls the Kubernetes cluster manager 67. kubectl â Controls the Kubernetes cluster manager â kubectl get pods or minions 68. kubectl â Controls the Kubernetes cluster manager â kubectl get pods or minions â kubectl create -f 69. kubectl â Controls the Kubernetes cluster manager â kubectl get pods or minions â kubectl create -f â kubectl update or delete 70. kubectl â Controls the Kubernetes cluster manager â kubectl get pods or minions â kubectl create -f â kubectl update or delete â kubectl resize âreplicas=3 replicationcontrollers 71. export KUBERNETES_PROVIDER=vagrant ./cluster/kube-up.sh Mac OS X Kubernetes (Vagrant) Master Minion 72. Recipe #2.1 Mac OS X Kubernetes (Vagrant) Master Minion Pod Docker
 (WildFly) http://blog.arungupta.me/javaee7-wildfly-kubernetes-mac-vagrant/ 73. Services â Abstract a set of pods as a single IP and port â Simple TCP/UDP load balancing â Creates environment variables in other pods â Like âDocker linksâ but across hosts â Stable endpoint for pods to reference â Allows list of pods to change dynamically 74. Recipe #2.2 Minion Pod Docker
 (WildFly) Pod Docker
 (MySQL) MySQL Service http://blog.arungupta.me/mysql-kubernetes-service-access-wildfly-pod-techtip72/ 75. Minion 2 Recipe #2.3 Minion 1 Pod Docker
 (WildFly) Pod Docker
 (MySQL) MySQL Service 76. Replication Controller 77. Replication Controller â Ensures specified number of pod âreplicasâ are running 78. Replication Controller â Ensures specified number of pod âreplicasâ are running â Pod templates are cookie cutters 79. Replication Controller â Ensures specified number of pod âreplicasâ are running â Pod templates are cookie cutters â Rescheduling 80. Replication Controller â Ensures specified number of pod âreplicasâ are running â Pod templates are cookie cutters â Rescheduling â Manual or auto-scale replicas 81. Replication Controller â Ensures specified number of pod âreplicasâ are running â Pod templates are cookie cutters â Rescheduling â Manual or auto-scale replicas â Rolling updates 82. Recipe #2.4 83. Recipe #2.4 84. Recipe #2.4 Minion 2 Minion 1 Pod Docker
 (WildFly) Pod Docker
 (MySQL) MySQL Service Pod Docker
 (WildFly) WildFly Service 85. Recipe #2.4 Minion 2 Minion 1 Pod Docker
 (WildFly) Pod Docker
 (MySQL) MySQL Service Pod Docker
 (WildFly) WildFly Service 86. Kubernetes: Pros and Cons â PROS â Manage related Docker containers as a unit â Container communication across hosts â Availability and scalability through automated deployment and monitoring of pods and their replicas, across hosts 87. Kubernetes: Pros and Cons â CONS â Lifecycle of applications - build, deploy, manage, promote â Port existing source code to run in Kubernetes â DevOps: Dev -> Test -> Production â No multi-tenancy â On-premise (available on GCE) â Assumes inter-pod networking as part of infrastructure â Requires explicit load balancer 88. Pod 7 ActiveMQ Pod 8 ActiveMQ âmqâ port 8161 port 8161 Pod 1 Apache Pod 2 Apache âwebâ port 80 port 80 Pod 5 MySQL Pod 6 MySQL âdbâ port 3306 port 3306 Pod 3 JBoss Pod 4 JBoss âjavaeeâ port 8080 port 8080 89. Pod 7 ActiveMQ Pod 8 ActiveMQ âmqâ port 8161 port 8161 Pod 1 Apache Pod 2 Apache âwebâ port 80 port 80 Pod 5 MySQL Pod 6 MySQL âdbâ port 3306 port 3306 Pod 3 JBoss Pod 4 JBoss âjavaeeâ port 8080 port 8080 90. Container Host Container Cluster Management User Experience 91. OpenShift 3 Features 92. OpenShift 3 Features â Push to production - full DevOps 93. OpenShift 3 Features â Push to production - full DevOps â Client tools for building web applications 94. OpenShift 3 Features â Push to production - full DevOps â Client tools for building web applications â Centralized administration and management of application component libraries 95. OpenShift 3 Features â Push to production - full DevOps â Client tools for building web applications â Centralized administration and management of application component libraries â Team and user isolation of containers, builds, and network communication in an easy multi-tenancy system 96. Recipe #3.1 â Start OpenShift as Docker container
 
 
 â Or run natively â Use osc (OpenShift Client) instead of kubectl with Kubernetes configuration file 97. Recipe #3.2 â (Alpha) tools generate project JSON configuration file that provide build/deployment 98. 40 99. Recipe #3.3 â Integration with JBoss Developer Studio (cooking) 100. Summary â Container runtime and image distribution â Roll your own solutions for everything â Runtime and operational management of containers
 â Lifecycle of applications - build, deploy, manage, promote â Manage tens of thousands of applications with teams 101. References â blog.arungupta.me/topics/containers/ â github.com/openshift/origin https://www.friendbookmark.com/videos/969/docker-kubernetes-istioUnderstanding Docker, Kubernetes and Service Mesh using Istio. Topics covered in this presentation slides: 1. ARAF KARSH HAMID Co-Founder / CTO MetaMagic Global Inc., NJ, USA @arafkarsh arafkarsh http://www.metamagicglobal.com Kind 2. Docker / Kubernetes / Istio Containers Container Orchestration Service Mesh 3. â 12 Factor App Methodology â Docker Concepts â Images and Containers â Anatomy of a Dockerfile â Networking / Volume Docker1 â Kubernetes Concepts â Namespace â Pods â RelicaSet â Deployment â Service / Endpoints â Ingress â Rollout and Undo â Auto Scale Kubernetes2 Day 1 - Basic 3 â API Gateway â Load Balancer â Service Discovery â Config Server â Circuit Breaker â Service Aggregator Infrastructure Design Patterns4 â Environment â Config Map â Pod Presets â Secrets 3 Kubernetes â Container App Setup â Hello World App â Multi Version Rollouts â Auto Scaling App 1 - HelloWorld2 4. Day 2 â Kubernetes Advanced Networking, Volumes, Logging & Helm Charts 4 â Docker / Kubernetes Networking â Pod to Pod Networking â Pod to Service Networking â Ingress and Egress â Internet Kubernetes Networking â Packet Path5 â Kubernetes IP Network â OSI | L2/3/7 | IP Tables | IP VS | BGP | VXLAN â Kube DNS | Proxy â LB, Cluster IP, Node Port â Ingress Controller Kubernetes Networking Advanced6 â Helm Charts Concepts â Package Charts â Install / Uninstall charts â Manage Release Cycles Helm Charts14 â In-Tree & Out-of-Tree Volume Plugins â Container Storage Interface â CSI â Volume Life Cycle â Persistent Volume â Persistent Volume Claims â Storage Class Kubernetes Volumes11 â Logging â Distributed Tracing â Jagger / Grafana / Prometheus Logging & Monitoring13 â Product App with Product Review Microservice App 2 â Product App with Multiple Versions6 5. â Jobs / Cron Jobs â Quotas / Limits / QoS â Pod / Node Affinity â Pod Disruption Budget â Kubernetes Commands Kubernetes Advanced Concepts12 Day 3 â Network Security, Service Mesh and Best Practices 5 â Docker Best Practices â Kubernetes Best Practices â Security Best Practices 15 Best Practices â Istio Concepts / Sidecar Pattern â Envoy Proxy / Cilium Integration 8 Service Mesh â Istio â Security â RBAC â Mesh Policy | Policy â Cluster RBAC Config â Service Role / Role Binding Istio â Security and RBAC10 â Gateway / Virtual Service â Destination Rule / Service Entry â AB Testing using Canary â Beta Testing using Canary Istio Traffic Management9 â Network Policy L3 / L4 â Security Policy for Microservices â Weave / Calico / Cilium / Flannel Kubernetes Network Security Policies7 â Shopping Portal App with 6 Microservices implementation. App 3 â Shopping Portal9 6. 12 Factor App Methodology 19-11-2019 6 4 Backing Services Treat Backing services like DB, Cache as attached resources 5 Build, Release, Run Separate Build and Run Stages 6 Process Execute App as One or more Stateless Process 7 Port Binding Export Services with Specific Port Binding 8 Concurrency Scale out via the process Model 9 Disposability Maximize robustness with fast startup and graceful exit 10 Dev / Prod Parity Keep Development, Staging and Production as similar as possible 11 Logs Treat logs as Event Streams 12 Admin Process Run Admin Tasks as one of Process Source:https://12factor.net/ Factors Description 1 Codebase One Code base tracked in revision control 2 Dependencies Explicitly declare dependencies 3 Configuration Configuration driven Apps 1 7. High Level Objectives 7 1. Create Docker Images 2. Run Docker Containers for testing. 3. Push the Containers to registry 4. Docker image as part of your Code Pipeline Process. 1. Create Pods (Containers) with Deployments 2. Create Services 3. Create Traffic Rules (Ingress / Gateway / Virtual Service / Destination Rules) 4. Create External Services From Creating a Docker Container to Deploying the Container in Production Kubernetes Cluster. All other activities revolves around these 8 points mentioned below. 1 #01 Slide Noâs #22 #22 #22 #40-54 #57 #136-144 #55 #145 8. Docker Containers Understanding Containers Docker Images / Containers Docker Networking 8 9. Whatâs a Container? Virtual Machine Looks like a Walks like a Runs like a 19-11-2019 9 Containers are a Sandbox inside Linux Kernel sharing the kernel with separate Network Stack, Process Stack, IPC Stack etc. 1 10. Servers / Virtual Machines / Containers Hardware OS BINS / LIB App 1 App 2 App 3 Server Hardware Host OS HYPERVISOR App 1 App 2 App 3 Guest OS BINS / LIB Guest OS BINS / LIB Guest OS BINS / LIB Type 1 Hypervisor Hardware Host OS App 1 App 2 App 3 BINS / LIB BINS / LIB BINS / LIB Container Hardware HYPERVISOR App 1 App 2 App 3 Guest OS BINS / LIB Guest OS BINS / LIB Guest OS BINS / LIB Type 2 Hypervisor 101 11. Docker containers are Linux Containers CGROUPS NAME SPACES Copy on Write DOCKER CONTAINER â Kernel Feature â Groups Processes â Control Resource Allocation â CPU, CPU Sets â Memory â Disk â Block I/O â Images â Not a File System â Not a VHD â Basically a tar file â Has a Hierarchy â Arbitrary Depth â Fits into Docker Registry â The real magic behind containers â It creates barriers between processes â Different Namespaces â PID Namespace â Net Namespace â IPC Namespace â MNT Namespace â Linux Kernel Namespace introduced between kernel 2.6.15 â 2.6.26 docker runlxc-start 11 https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/resource_management_guide/ch01 19-11-2019 1 12. Docker Container â Linux and Windows Control Groups cgroups Namespaces Pid, net, ipc, mnt, uts Layer Capabilities Union File Systems: AUFS, btrfs, vfs Control Groups Job Objects Namespaces Object Namespace, Process Table. Networking Layer Capabilities Registry, UFS like extensions Namespaces: Building blocks of the Containers 121 13. Docker Key Concepts â Docker images â A Docker image is a read-only template. â For example, an image could contain an Ubuntu operating system with Apache and your web application installed. â Images are used to create Docker containers. â Docker provides a simple way to build new images or update existing images, or you can download Docker images that other people have already created. â Docker images are the build component of Docker. â Docker containers â Docker containers are similar to a directory. â A Docker container holds everything that is needed for an application to run. â Each container is created from a Docker image. â Docker containers can be run, started, stopped, moved, and deleted. â Each container is an isolated and secure application platform. â Docker containers are the run component of Docker. â Docker Registries â Docker registries hold images. â These are public or private stores from which you upload or download images. â The public Docker registry is called Docker Hub. â It provides a huge collection of existing images for your use. â These can be images you create yourself or you can use images that others have previously created. â Docker registries are the distribution component of Docker. 13 Images Containers 19-11-2019 14. Docker DaemonDocker Client How Docker worksâ. $ docker search â. $ docker build â. $ docker container create .. Docker Hub Images Containers $ docker container run .. $ docker container start .. $ docker container stop .. $ docker container ls .. $ docker push â. $ docker swarm .. 19-11-2019 14 21 34 1. Search for the Container 2. Docker Daemon Sends the request to Hub 3. Downloads the image 4. Run the Container from the image 1 15. Linux Kernel 19-11-2019 15 HOST OS (Ubuntu) Client Docker Daemon Cent OS Alpine Debian HostLinuxKernel Host Kernel Host Kernel Host KernelAll the containers will have the same Host OS Kernel If you require a specific Kernel version then Host Kernel needs to be updated 1 16. Windows Kernel 19-11-2019 16 HOST OS (Windows 10) Client Docker Daemon Nano Server Server Core Nano Server WindowsKernel Host Kernel Host Kernel Host KernelAll the containers will have the same Host OS Kernel If you require a specific Kernel version then Host Kernel needs to be updated 1 17. Docker Image structure â Images are read-only. â Multiple layers of image gives the final Container. â Layers can be sharable. â Layers are portable. â Debian Base image â Emacs â Apache â Writable Container 19-11-2019 171 18. Running a Docker Container $ ID=$(docker container run -d ubuntu /bin/bash -c âwhile true; do date; sleep 1; doneâ) Creates a Docker Container of Ubuntu OS and runs the container and execute bash shell with a script. $ docker container logs $ID Shows output from the( bash script) container $ docker container ls List the running Containers $ docker pull ubuntu Docker pulls the image from the Docker Registry 19-11-2019 181 When you copy the commands for testing change â quotes to proper quotes. Microsoft PowerPoint messes with the quotes. 19. Anatomy of a Dockerfile Command Description Example FROM The FROM instruction sets the Base Image for subsequent instructions. As such, a valid Dockerfile must have FROM as its first instruction. The image can be any valid image â it is especially easy to start by pulling an image from the Public repositories FROM ubuntu FROM alpine MAINTAINER The MAINTAINER instruction allows you to set the Author field of the generated images. MAINTAINER johndoe LABEL The LABEL instruction adds metadata to an image. A LABEL is a key-value pair. To include spaces within a LABEL value, use quotes and blackslashes as you would in command-line parsing. LABEL version="1.0â LABEL vendor=âM2â RUN The RUN instruction will execute any commands in a new layer on top of the current image and commit the results. The resulting committed image will be used for the next step in the Dockerfile. RUN apt-get install -y curl ADD The ADD instruction copies new files, directories or remote file URLs from and adds them to the filesystem of the container at the path . ADD hom* /mydir/ ADD hom?.txt /mydir/ COPY The COPY instruction copies new files or directories from and adds them to the filesystem of the container at the path . COPY hom* /mydir/ COPY hom?.txt /mydir/ ENV The ENV instruction sets the environment variable to the value . This value will be in the environment of all "descendent" Dockerfile commands and can be replaced inline in many as well. ENV JAVA_HOME /JDK8 ENV JRE_HOME /JRE8 19 19-11-2019 1 20. Anatomy of a Dockerfile Command Description Example VOLUME The VOLUME instruction creates a mount point with the specified name and marks it as holding externally mounted volumes from native host or other containers. The value can be a JSON array, VOLUME ["/var/log/"], or a plain string with multiple arguments, such as VOLUME /var/log or VOLUME /var/log VOLUME /data/webapps USER The USER instruction sets the user name or UID to use when running the image and for any RUN, CMD and ENTRYPOINT instructions that follow it in the Dockerfile. USER johndoe WORKDIR The WORKDIR instruction sets the working directory for any RUN, CMD, ENTRYPOINT, COPY and ADD instructions that follow it in the Dockerfile. WORKDIR /home/user CMD There can only be one CMD instruction in a Dockerfile. If you list more than one CMD then only the last CMD will take effect. The main purpose of a CMD is to provide defaults for an executing container. These defaults can include an executable, or they can omit the executable, in which case you must specify an ENTRYPOINT instruction as well. CMD echo "This is a test." | wc - EXPOSE The EXPOSE instructions informs Docker that the container will listen on the specified network ports at runtime. Docker uses this information to interconnect containers using links and to determine which ports to expose to the host when using the âP flag with docker client. EXPOSE 8080 ENTRYPOINT An ENTRYPOINT allows you to configure a container that will run as an executable. Command line arguments to docker run will be appended after all elements in an exec form ENTRYPOINT, and will override all elements specified using CMD. This allows arguments to be passed to the entry point, i.e., docker run -d will pass the -d argument to the entry point. You can override the ENTRYPOINT instruction using the docker run --entrypoint flag. ENTRYPOINT ["top", "-b"] 19-11-2019 201 21. 19-11-2019 21 Docker Image â Dockerfile â Docker Container Management â Docker Images 1 22. Build Docker Containers as easy as 1-2-3 19-11-2019 22 Create Dockerfile 1 Build Image 2 Run Container 3 1 23. Build a Docker Java image 1. Create your Dockerfile â FROM â RUN â ADD â WORKDIR â USER â ENTRYPOINT 2. Build the Docker image 3. Run the Container $ docker build -t org/java:8 . 231 $ docker container run âit org/java:8 24. Docker Container Management $ ID=$(docker container run âd ubuntu /bin/bash) $ docker container stop $ID Start the Container and Store ID in ID field Stop the container using Container ID $ docker container stop $(docker container ls âaq) Stops all the containers $ docker container rm $ID Remove the Container $ docker container rm $(docker container ls âaq) Remove ALL the Container (in Exit status) $ docker container prune Remove ALL stopped Containers) $ docker container run ârestart=Policy âd âit ubuntu /sh Policies = NO / ON-FAILURE / ALWAYS $ docker container run ârestart=on-failure:3 âd âit ubuntu /sh Will re-start container ONLY 3 times if a failure happens $ docker container start $ID Start the container 19-11-2019 241 25. Docker Container Management $ ID=$(docker container run âd -i ubuntu) $ docker container exec -it $ID /bin/bash Start the Container and Store ID in ID field Inject a Process into Running Container $ ID=$(docker container run âd âi ubuntu) $ docker container exec inspect $ID Start the Container and Store ID in ID field Read Containers MetaData $ docker container run âit ubuntu /bin/bash # apt-get update # apt-get installây apache2 # exit $ docker container ls âa $ docker container commit âauthor=ânameâ â message=âUbuntu / Apache2â containerId apache2 Docker Commit â Start the Ubuntu Container â Install Apache â Exit Container â Get the Container ID (Ubuntu) â Commit the Container with new name $ docker container run âcap-drop=chown âit ubuntu /sh To prevent Chown inside the Container 19-11-2019 251 Source: https://github.com/meta-magic/kubernetes_workshop 26. Docker Image Commands $ docker login â. Log into the Docker Hub to Push images $ docker push image-name Push the image to Docker Hub $ docker image history image-name Get the History of the Docker Image $ docker image inspect image-name Get the Docker Image details $ docker image save âoutput=file.tar image-name Save the Docker image as a tar ball. $ docker container export âoutput=file.tar c79aa23dd2 Export Container to file. 19-11-2019 261 Source: https://github.com/meta-magic/kubernetes_workshop $ docker image rm image-name Remove the Docker Image $ docker rmi $(docker images | grep '^' | tr -s " " | cut -d " " -f 3) 27. Build Docker Apache image 1. Create your Dockerfile â FROM alpine â RUN â COPY â EXPOSE â ENTRYPOINT 2. Build the Docker image 3. Run the Container $ docker build -t org/apache2 . $ docker container run âd âp 80:80 org/apache2 $ curl localhost 19-11-2019 271 28. Build Docker Tomcat image 1. Create your Dockerfile â FROM alpine â RUN â COPY â EXPOSE â ENTRYPOINT 2. Build the Docker image 3. Run the Container $ docker build -t org/tomcat . $ docker container run âd âp 8080:8080 org/tomcat $ curl localhost:8080 19-11-2019 281 29. Docker Images in the Github Workshop Ubuntu JRE 8 JRE 11 Tomcat 8 Tomcat 9 My App 1 Tomcat 9 My App 3 Spring Boot My App 4 From Ubuntu Build My Ubuntu From My Ubuntu Build My JRE8 From My Ubuntu Build My JRE11 From My JRE 11 Build My Boot From My Boot Build My App 4 From My JRE8 Build My TC8 From My TC8 Build My App 1 19-11-2019 29 My App 2 1 Source: https://github.com/meta-magic/kubernetes_workshop 30. Docker Images in the Github Workshop Alpine Linux JRE 8 JRE 11 Tomcat 8 Tomcat 9 My App 1 Tomcat 9 My App 3 Spring Boot My App 4 From Alpine Build My Alpine From My Alpine Build My JRE8 From My Alpine Build My JRE11 From My JRE 11 Build My Boot From My Boot Build My App 4 From My JRE8 Build My TC8 From My TC8 Build My App 1 19-11-2019 30 My App 2 1 Source: https://github.com/meta-magic/kubernetes_workshop 31. 19-11-2019 311 Docker Networking â Docker Networking â Bridge / Host / None â Docker Container sharing IP Address â Docker Communication â Node to Node â Docker Volumes 32. Docker Networking â Bridge / Host / None $ docker network ls $ docker container run --rm --network=host alpine brctl show $ docker network create tenSubnet âsubnet 10.1.0.0/16 19-11-2019 321 33. Docker Networking â Bridge / Host / None $ docker container run --rm -ânet=host alpine ip address$ docker container run --rm alpine ip address $ docker container run ârm ânet=none alpine ip address No Network Stack https://docs.docker.com/network/#network-drivers 19-11-2019 331 34. Docker Containers Sharing IP Address $ docker container run --name ipctr âitd alpine $ docker container run --rm --net container:ipctr alpine ip address IP (Container) Service 1 (Container) Service 3 (Container) Service 2 (Container) 19-11-2019 341 $ docker container exec ipctr ip address 35. Docker Networking: Node to Node Same IP Addresses for the Containers across different Nodes. This requires NAT. 351 Container 1 172.17.3.2 Web Server 8080 Veth: eth0 Container 2 172.17.3.3 Microservice 9002 Veth: eth0 Container 3 172.17.3.4 Microservice 9003 Veth: eth0 Container 4 172.17.3.5 Microservice 9004 Veth: eth0 IP tables rules eth0 10.130.1.101/24 Node 1 Docker0 Bridge 172.17.3.1/16 Veth0 Veth1 Veth2 Veth3 Container 1 172.17.3.2 Web Server 8080 Veth: eth0 Container 2 172.17.3.3 Microservice 9002 Veth: eth0 Container 3 172.17.3.4 Microservice 9003 Veth: eth0 Container 4 172.17.3.5 Microservice 9004 Veth: eth0 IP tables rules eth0 10.130.1.102/24 Node 2 Docker0 Bridge 172.17.3.1/16 Veth0 Veth1 Veth2 Veth3Veth: eth0 Veth0 Veth Pairs connected to the container and the Bridge 36. Docker Volumes $ docker volume create hostvolume Data Volumes are special directory in the Docker Host. $ docker volume ls $ docker container run âit ârm âv hostvolume:/data alpine # echo âThis is a test from the Containerâ > /data/data.txt 19-11-2019 361 Source:https://github.com/meta-magic/kubernetes_workshop 37. Docker Volumes $ docker container run - - rm âv $HOME/data:/data alpine Mount Specific File Path 19-11-2019 371 Source:https://github.com/meta-magic/kubernetes_workshop 38. Kubernetes 19-11-2019 38 39. Deployment â Updates and rollbacks, Canary Release D ReplicaSet â Self Healing, Scalability, Desired State R Worker Node 1 Master Node (Control Plane) Kubernetes Architecture POD POD itself is a Linux Container, Docker container will run inside the POD. PODs with single or multiple containers (Sidecar Pattern) will share Cgroup, Volumes, Namespaces of the POD. (Cgroup / Namespaces) Scheduler Controller Manager Using yaml or json declare the desired state of the app. State is stored in the Cluster store. Self healing is done by Kubernetes using watch loops if the desired state is changed. POD POD POD BE 1.210.1.2.34 BE 1.210.1.2.35 BE 1.210.1.2.36 BE 15.1.2.100 DNS: a.b.com 1.2 Service Pod IP Address is dynamic, communication should be based on Service which will have routable IP and DNS Name. Labels (BE, 1.2) play a critical role in ReplicaSet, Deployment, & Services etc. Cluster Store etcd Key Value Store Pod Pod Pod Label Selector selects pods based on the Labels. Label Selector Label Selector Label Selector Node Controller End Point Controller Deployment Controller Pod Controller â. Labels Internet Firewall K8s Cluster Cloud Controller For the cloud providers to manage nodes, services, routes, volumes etc. Kubelet Node Manager Container Runtime Interface Port 10255 gRPC ProtoBuf Kube-Proxy Network Proxy TCP / UDP Forwarding IPTABLES / IPVS Allows multiple implementation of containers from v1.7 RESTful yaml / json $ kubectl â. Port 443API Server Pod IP ...34 ...35 ...36EP â Declarative Model â Desired State Key Aspects Namespace1Namespace2 â Pods â ReplicaSet â Deployment â Service â Endpoints â StatefulSet â Namespace â Resource Quota â Limit Range â Persistent Volume Kind Secrets Kind â apiVersion: â kind: â metadata: â spec: Declarative Model â Pod â ReplicaSet â Service â Deployment â Virtual Service â Gateway, SE, DR â Policy, MeshPolicy â RbaConfig â Prometheus, Rule, â ListChekcer â @ @ Annotations Names Cluster IP Node Port Load Balancer External Name @ Ingress 392 40. 40 Focus on the Declarative Model 2 41. Ubuntu Installation Kubernetes Setup â Minikube $ sudo snap install kubectl --classic 19-11-2019 41 Install Kubectl using Snap Package Manager $ kubectl version Shows the Current version of Kubectl â Minikube provides a developer environment with master and a single node installation within the Minikube with all necessary add-ons installed like DNS, Ingress controller etc. â In a real world production environment you will have master installed (with a failover) and ânâ number of nodes in the cluster. â If you go with a Cloud Provider like Amazon EKS then the node will be created automatically based on the load. â Minikube is available for Linux / Mac OS and Windows. $ curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.30.0/minikube-linux-amd64 $ chmod +x minikube && sudo mv minikube /usr/local/bin/ https://kubernetes.io/docs/tasks/tools/install-kubectl/ 2 Source: https://github.com/meta-magic/kubernetes_workshop 42. Windows Installation Kubernetes Setup â Minikube C:> choco install kubernetes-cli 19-11-2019 42 Install Kubectl using Choco Package Manager C:> kubectl version Shows the Current version of Kubectl Mac OS Installation $ brew install kubernetes-cli Install Kubectl using brew Package Manager $ kubectl version Shows the Current version of Kubectl C:> cd c:usersyouraccount C:> mkdir .kube Create .kube directory $ curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-darwin-amd64 $ chmod +x minikube && sudo mv minikube /usr/local/bin/ C:> minikube-installer.exe Install Minikube using Minikube Installer https://kubernetes.io/docs/tasks/tools/install-kubectl/ 2 Source:https://github.com/meta-magic/kubernetes_workshop $ brew update; brew cask install minikube Install Minikube using Homebrew or using curl 43. Kubernetes Minikube - Commands 43 Commands $ minikube status Shows the status of minikube installation $ minikube start Start minikube 2 All workshop examples Source Code: https://github.com/meta-magic/kubernetes_workshop $ minikube stop Stop Minikube $ minikube ip Shows minikube IP Address $ minikube addons list Shows all the addons $ minikube addons enable ingress Enable ingress in minikube $ minikube start --memory=8192 --cpus=4 --kubernetes-version=1.14.2 8 GB RAM and 4 Cores $ minikube dashboard Access Kubernetes Dashboard in minikube $ minikube start --network-plugin=cni --extra-config=kubelet.network-plugin=cni --memory=5120 With Cilium Network Driver $ kubectl create -n kube-system -f https://raw.githubusercontent.com/cilium/cilium/v1.3/examples/kubernetes/addons/etcd/standalone-etcd.yaml $ kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.3/examples/kubernetes/1.12/cilium.yaml 44. K8s Setup â Master / Nodes : On Premise 442 Cluster Machine Setup 1. Switch off Swap 2. Set Static IP to Network interface 3. Add IP to Host file $ k8s-1-cluster-machine-setup.sh 4. Install Docker 5. Install Kubernetes Run the cluster setup script to install the Docker and Kubernetes in all the machines (master and worker node) 1 Master Setup Setup kubernetes master with pod network 1. Kubeadm init 2. Install CNI Driver $ k8s-2-master-setup.sh $ k8s-3-cni-driver-install.sh $ k8s-3-cni-driver-uninstall.sh $ kubectl get po --all-namespaces Check Driver Pods Uninstall the driver 2 Node Setup n1$ kubeadm join --token t IP:Port Add the worker node to Kubernetes Master $ kubectl get nodes Check Events from namespace 3 $ kubectl get events ân namespace Check all the nodes $ sudo ufw enable $ sudo ufw allow 31100 Source Code: https://github.com/meta-magic/metallb-baremetal-example Only if the Firewall is blocking your Pod Al the above-mentioned shell scripts are available in the Source Code Repository $ sudo ufw allow 443 45. Kubernetes Setup â Master / Nodes $ kubeadm init node1$ kubeadm join --token enter-token-from-kubeadm-cmd Node-IP:Port Adds a Node $ kubectl get nodes $ kubectl cluster-info List all Nodes $ kubectl run hello-world --replicas=7 --labels="run=load-balancer-example" --image=metamagic/hello:1.0 --port=8080 Creates a Deployment Object and a ReplicaSet object with 7 replicas of Hello-World Pod running on port 8080 $ kubectl expose deployment hello-world --type=LoadBalancer --name=hello-world-service List all the Hello-World Deployments$ kubectl get deployments hello-world Describe the Hello-World Deployments$ kubectl describe deployments hello-world List all the ReplicaSet$ kubectl get replicasets Describe the ReplicaSet$ kubectl describe replicasets List the Service Hello-World-Service with Custer IP and External IP $ kubectl get services hello-world-service Describe the Service Hello-World-Service$ kubectl describe services hello-world-service Creates a Service Object that exposes the deployment (Hello-World) with an external IP Address. List all the Pods with internal IP Address $ kubectl get pods âo wide $ kubectl delete services hello-world-service Delete the Service Hello-World-Service $ kubectl delete deployment hello-world Delete the Hello-Word Deployment Create a set of Pods for Hello World App with an External IP Address (Imperative Model) Shows the cluster details $ kubectl get namespace Shows all the namespaces $ kubectl config current-context Shows Current Context 452 Source: https://github.com/meta-magic/kubernetes_workshop 46. 3 Fundamental Concepts 1. Desired State 2. Current State 3. Declarative Model 19-11-2019 462 47. Kubernetes Workload Portability 47 Goals 1. Abstract away Infrastructure Details 2. Decouple the App Deployment from Infrastructure (On-Premise or Cloud) To help Developers 1. Write Once, Run Anywhere (Workload Portability) 2. Avoid Vendor Lock-In Cloud On-Premise 2 48. 19-11-2019 48 Kubernetes Getting Started â Namespace â Pods / ReplicaSet / Deployment â Service / Endpoints â Ingress â Rollout / Undo â Auto Scale 2 49. Kubernetes Commands â Namespace (Declarative Model) $ kubectl config set-context $(kubectl config current-context) --namespace=your-ns The above command will let you switch the namespace to your namespace (your-ns). $ kubectl get namespace $ kubectl describe ns ns-name $ kubectl create âf app-ns.yml List all the Namespaces Describe the Namespace Create the Namespace $ kubectl apply âf app-ns.yml Apply the changes to the Namespace $ kubectl get pods ânamespace= ns-name List the Pods from your namespace â Namespaces are used to group your teams and softwareâs in logical business group. â A definition of Service will add a entry in DNS with respect to Namespace. â Not all objects are there in Namespace. Ex. Nodes, Persistent Volumes etc. 19-11-2019 492 50. â Pod is a shared environment for one of more Containers. â Pod in a Kubernetes cluster has a unique IP address, even Pods on the same Node. â Pod is a pause Container Kubernetes Pods $ kubectl create âf app1-pod.yml $ kubectl get pods Atomic Unit ContainerPodVirtual Server SmallBig 19-11-2019 502 Source: https://github.com/meta-magic/kubernetes_workshop 51. Kubernetes Commands â Pods (Declarative Model) $ kubectl exec pod-name ps aux $ kubectl exec âit pod-name sh $ kubectl exec âit âcontainer container-name pod-name sh By default kubectl executes the commands in the first container in the pod. If you are running multiple containers (sidecar pattern) then you need to pass âcontainer flag and give the name of the container in the Pod to execute your command. You can see the ordering of the containers and its name using describe command. $ kubectl get pods $ kubectl describe pods pod-name $ kubectl get pods -o json pod-name $ kubectl create âf app-pod.yml List all the pods Describe the Pod details List the Pod details in JSON format Create the Pod Execute commands in the first Container in the Pod Log into the Container Shell $ kubectl get pods -o wide List all the Pods with Pod IP Addresses $ kubectl apply âf app-pod.yml Apply the changes to the Pod $ kubectl replace âf app-pod.yml Replace the existing config of the Pod $ kubectl describe pods âl app=name Describe the Pod based on the label value 19-11-2019 512 $ kubectl logs pod-name container-name Source: https://github.com/meta-magic/kubernetes_workshop 52. â Pods wrap around containers with benefits like shared location, secrets, networking etc. â ReplicaSet wraps around Pods and brings in Replication requirements of the Pod â ReplicaSet Defines 2 Things â Pod Template â Desired No. of Replicas Kubernetes ReplicaSet What we want is the Desired State. Game On! 19-11-2019 522 Source: https://github.com/meta-magic/kubernetes_workshop 53. Kubernetes Commands â ReplicaSet (Declarative Model) $ kubectl delete rs/app-rs cascade=false $ kubectl get rs $ kubectl describe rs rs-name $ kubectl get rs/rs-name $ kubectl create âf app-rs.yml List all the ReplicaSets Describe the ReplicaSet details Get the ReplicaSet status Create the ReplicaSet which will automatically create all the Pods Deletes the ReplicaSet. If the cascade=true then deletes all the Pods, Cascade=false will keep all the pods running and ONLY the ReplicaSet will be deleted. $ kubectl apply âf app-rs.yml Applies new changes to the ReplicaSet. For example Scaling the replicas from x to x + new value. 19-11-2019 532 54. Kubernetes Commands â Deployment (Declarative Model) â Deployments manages ReplicaSets and â ReplicaSets manages Pods â Deployment is all about Rolling updates and â Rollbacks â Canary Deployments 19-11-2019 542 Source: https://github.com/meta-magic/kubernetes_workshop 55. Kubernetes Commands â Deployment (Declarative Model) List all the Deployments Describe the Deployment details Show the Rollout status of the Deployment Creates Deployment Deployments contains Pods and its Replica information. Based on the Pod info Deployment will start downloading the containers (Docker) and will install the containers based on replication factor. Updates the existing deployment. Show Rollout History of the Deployment $ kubectl get deploy app-deploy $ kubectl describe deploy app-deploy $ kubectl rollout status deployment app-deploy $ kubectl rollout history deployment app-deploy $ kubectl create âf app-deploy.yml $ kubectl apply âf app-deploy.yml --record $ kubectl rollout undo deployment app-deploy - -to-revision=1 $ kubectl rollout undo deployment app-deploy - -to-revision=2 Rolls back or Forward to a specific version number of your app. $ kubectl scale deployment app-deploy - -replicas=6 Scale up the pods to 6 from the initial 2 Pods. 552 Source: https://github.com/meta-magic/kubernetes_workshop 56. Kubernetes Services Why do we need Services? â Accessing Pods from Inside the Cluster â Accessing Pods from Outside â Autoscale brings Pods with new IP Addresses or removes existing Pods. â Pod IP Addresses are dynamic. Service will have a stable IP Address. Service uses Labels to associate with a set of Pods Service Types 1. Cluster IP (Default) 2. Node Port 3. Load Balancer 4. External Name19-11-2019 562 Source: https://github.com/meta-magic/kubernetes_workshop 57. Kubernetes Commands â Service / Endpoints (Declarative Model) $ kubectl delete svc app-service $ kubectl create âf app-service.yml List all the Services Describe the Service details List the status of the Endpoints Create a Service for the Pods. Service will focus on creating a routable IP Address and DNS for the Pods Selected based on the labels defined in the service. Endpoints will be automatically created based on the labels in the Selector. Deletes the Service. $ kubectl get svc $ kubectl describe svc app-service $ kubectl get ep app-service $ kubectl describe ep app-service Describe the Endpoint Details  Cluster IP (default) - Exposes the Service on an internal IP in the cluster. This type makes the Service only reachable from within the cluster.  Node Port - Exposes the Service on the same port of each selected Node in the cluster using NAT. Makes a Service accessible from outside the cluster using :. Superset of ClusterIP.  Load Balancer - Creates an external load balancer in the current cloud (if supported) and assigns a fixed, external IP to the Service. Superset of NodePort.  External Name - Exposes the Service using an arbitrary name (specified by external Name in the spec) by returning a CNAME record with the name. No proxy is used. This type requires v1.7 or higher of kube-dns. 19-11-2019 572 Source: https://github.com/meta-magic/kubernetes_workshop 58. Kubernetes Ingress (Declarative Model) An Ingress is a collection of rules that allow inbound connections to reach the cluster services. Ingress is still a beta feature in Kubernetes Ingress Controllers are Pluggable. Ingress Controller in AWS is linked to AWS Load Balancer. Source: https://kubernetes.io/docs/concepts/services-networking/ingress/#ingress-controllers 19-11-2019 582 Source: https://github.com/meta-magic/kubernetes_workshop 59. Kubernetes Ingress (Declarative Model) An Ingress is a collection of rules that allow inbound connections to reach the cluster services. Ingress is still a beta feature in Kubernetes Ingress Controllers are Pluggable. Ingress Controller in AWS is linked to AWS Load Balancer. Source: https://kubernetes.io/docs/concepts/services-networking/ingress/#ingress-controllers 19-11-2019 592 60. Kubernetes Auto Scaling Pods (Declarative Model) â You can declare the Auto scaling requirements for every Deployment (Microservices). â Kubernetes will add Pods based on the CPU Utilization automatically. â Kubernetes Cloud infrastructure will automatically add Nodes if it ran out of available Nodes. CPU utilization kept at 10% to demonstrate the auto scaling feature. Ideally it should be around 80% - 90% 19-11-2019 602 Source: https://github.com/meta-magic/kubernetes_workshop 61. Kubernetes Horizontal Pod Auto Scaler $ kubectl autoscale deployment appname --cpu-percent=50 --min=1 --max=10 $ kubectl run -it podshell --image=metamagicglobal/podshell Hit enter for command prompt $ while true; do wget -q -O- http://yourapp.default.svc.cluster.local; done Deploy your app with auto scaling parameters Generate load to see auto scaling in action $ kubectl get hpa $ kubectl attach podshell-name -c podshell -it To attach to the running container 612 Source: https://github.com/meta-magic/kubernetes_workshop 62. Auto Scaling - Advanced (Declarative Model) CPU utilization kept at 10% to demonstrate the auto scaling feature. Ideally it should be around 80% - 90%19-11-2019 622 63. 19-11-2019 63 Kubernetes App Setup â Environment â Config Map â Pod Preset â Secrets 3 64. Detach the Configuration information of the App from the Container Image. Config Map lets you create multiple profiles for your Dev, QA and Prod environment. Config Map All the Database configurations like passwords, certificates, OAuth tokens, etc., can be stored in secrets. Secret Helps you create common configuration which can be injected to Pod based on a Criteria (selected using Label). For Ex. SMTP config, SMS config. Pod Preset Environment option let you pass any info to the pod thru Environment Variables. Environment Container App Setup 19-11-2019 643 65. Kubernetes Pod Environment Variables 19-11-2019 65 Source: https://github.com/meta-magic/kubernetes_workshop 3 66. Kubernetes Adding Config to Pod Config Maps allow you to decouple configuration artifacts from image content to keep containerized applications portable. Source: https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/ 19-11-2019 66 Source: https://github.com/meta-magic/kubernetes_workshop 3 67. Kubernetes Pod Presets A Pod Preset is an API resource for injecting additional runtime requirements into a Pod at creation time. You use label selectors to specify the Pods to which a given Pod Preset applies. Using a Pod Preset allows pod template authors to not have to explicitly provide all information for every pod. This way, authors of pod templates consuming a specific service do not need to know all the details about that service. Source: https://kubernetes.io/docs/concepts/workloads/pods/podpreset/ 19-11-2019 67 Source: https://github.com/meta-magic/kubernetes_workshop 3 68. Kubernetes Pod Secrets Objects of type secret are intended to hold sensitive information, such as passwords, OAuth tokens, and ssh keys. Putting this information in a secret is safer and more flexible than putting it verbatim in a pod definition or in a docker Source: https://kubernetes.io/docs/concepts/configuration/secret/ 19-11-2019 68 Source: https://github.com/meta-magic/kubernetes_workshop 3 69. 19-11-2019 694 Infrastructure Design Patterns â API Gateway â Load balancer â Service discovery â Circuit breaker â Service Aggregator â Let-it crash pattern 70. API Gateway Design Pattern â Software Stack UILayer WS BL DL Database Shopping Cart Order Customer Product Firewall Users API Gateway LoadBalancer CircuitBreaker UILayer WebServices BusinessLogic DatabaseLayer Product SE MySQL DB Product Microservice With 4 node cluster LoadBalancer CircuitBreaker UILayer WebServices BusinessLogic DatabaseLayer Customer Redis DB Customer Microservice With 2 node cluster Users Access the Monolithic App Directly API Gateway (Reverse Proxy Server) routes the traffic to appropriate Microservices (Load Balancers) 4 71. API Gateway â Kubernetes Implementation /customer /product /cart /order API Gateway Ingress Deployment / Replica / Pod NodesKubernetes Objects Firewall Customer Pod Customer Pod Customer Pod Customer Service N1 N2 N2 EndPoints Product Pod Product Pod Product Pod Product Service N4 N3 MySQL DB EndPoints Review Pod Review Pod Review Pod Review Service N4 N3 N1 Service Call Kube DNS EndPoints Internal Load Balancers Users Routing based on Layer 3,4 and 7 Redis DB Mongo DB Load Balancer 4 72. 72 API Gateway â Kubernetes / Istio /customer /product /auth /order API Gateway Virtual Service Deployment / Replica / Pod NodesIstio Sidecar - Envoy Load Balancer Firewall P M CIstio Control Plane MySQL Pod N4 N3 Destination Rule Product Pod Product Pod Product Pod Product Service Service Call Kube DNS EndPoints Internal Load Balancers 72 Kubernetes Objects Istio Objects Users Review Pod Review Pod Review Pod Review Service N1 N4 N3EndPoints Customer Pod Customer Pod Customer Pod Customer Service N1 N2 N2 Destination Rule EndPoints Redis DB Mongo DB 4 73. Load Balancer Design Pattern Firewall Users API Gateway Load Balancer CircuitBreaker UILayer WebServices BusinessLogic DatabaseLayer Product SE MySQL DB Product Microservice With 4 node cluster Load Balancer CB=Hystrix UILayer WebServices BusinessLogic DatabaseLayer Customer Redis DB Customer Microservice With 2 node cluster API Gateway (Reverse Proxy Server) routes the traffic to appropriate Microservices (Load Balancers) Load Balancer Rules 1. Round Robin 2. Based on Availability 3. Based on Response Time 4 74. Ingress Load Balancer â Kubernetes Model Kubernetes Objects Firewall Users Product 1 Product 2 Product 3 Product Service N4 N3 N1 EndPoints Internal Load Balancers DB Load Balancer API Gateway N1 N2 N2Customer 1 Customer 2 Customer 3 Customer Service EndPoints DB Internal Load Balancers Pods Nodes â Load Balancer receives the (request) packet from the User and it picks up a Virtual Machine in the Cluster to do the internal Load Balancing. â Kube Proxy using IP Tables redirect the Packet using internal load Balancing rules. â Packet enters Kubernetes Cluster and reaches Node (of that specific Pod) and Node handover the packet to the Pod. /customer /product /cart 4 75. Service Discovery â NetFlix Network Stack Model Firewall Users API Gateway LoadBalancer CircuitBreaker Product MySQL DB Product Microservice With 4 node cluster LoadBalancer CircuitBreaker UILayer WebServices BusinessLogic DatabaseLayer Customer Redis DB Customer Microservice With 2 node cluster â In this model Developers write the code in every Microservice to register with NetFlix Eureka Service Discovery Server. â Load Balancers and API Gateway also registers with Service Discovery. â Service Discovery will inform the Load Balancers about the instance details (IP Addresses). Service Discovery 4 76. Ingress Service Discovery â Kubernetes Model Kubernetes Objects Firewall Users Product 1 Product 2 Product 3 Product Service N4 N3 N1 EndPoints Internal Load Balancers DB API Gateway N1 N2 N2Customer 1 Customer 2 Customer 3 Customer Service EndPoints DB Internal Load Balancers Pods Nodes â API Gateway (Reverse Proxy Server) doesn't know the instances (IP Addresses) of News Pod. It knows the IP address of the Services defined for each Microservice (Customer / Product etc.) â Services handles the dynamic IP Addresses of the pods. Services Endpoints will automatically discover the new Pods based on Labels. Service Definition from Kubernetes Perspective /customer /product /cart Service Call Kube DNS 4 77. Circuit Breaker Pattern /ui /productms If Product Review is not available Product service will return the product details with a message review not available. Reverse Proxy Server Ingress Deployment / Replica / Pod NodesKubernetes Objects Firewall UI Pod UI Pod UI Pod UI Service N1 N2 N2 EndPoints Product Pod Product Pod Product Pod Product Service N4 N3 MySQL Pod EndPoints Internal Load Balancers 77 Users Routing based on Layer 3,4 and 7 Review Pod Review Pod Review Pod Review Service N4 N3 N1 Service Call Kube DNS EndPoints 4 78. Service Aggregator Pattern /newservice Reverse Proxy Server Ingress Deployment / Replica / Pod Nodes Kubernetes Objects Firewall Service Call Kube DNS Users Internal Load Balancers EndPoints News Pod News Pod News Pod News Service N4 N3 N2 News Service Portal â News Category wise Microservices â Aggregator Microservice to aggregate all category of news. Auto Scaling â Sports Events (IPL / NBA) spikes the traffic for Sports Microservice. â Auto scaling happens for both News and Sports Microservices. N1 N2 N2National National National National Service EndPoints Internal Load Balancers DB N1 N2 N2Politics Politics Politics Politics Service EndPoints DB Sports Sports Sports Sports Service N4 N3 N1 EndPoints Internal Load Balancers DB 4 79. Music UI 11/19/2019 79 Play Count Discography Albums 4 80. Service Aggregator Pattern /artist Reverse Proxy Server Ingress Deployment / Replica / Pod Nodes Kubernetes Objects Firewall Service Call Kube DNS 80 Users Internal Load Balancers EndPoints Artist Pod Artist Pod Artist Pod Artist Service N4 N3 N2 Spotify Microservices â Artist Microservice combines all the details from Discography, Play count and Playlists. Auto Scaling â Scaling of Artist and downstream Microservices will automatically scale depends on the load factor. N1 N2 N2Discography Discography Discography Discography Service EndPoints Internal Load Balancers DB N1 N2 N2Play Count Play Count Play Count Play Count Service EndPoints DB Playlist Playlist Playlist Playlist Service N4 N3 N1 EndPoints Internal Load Balancers DB 4 81. Config Store â Spring Config Server Firewall Users API Gateway LoadBalancer CircuitBreaker Product MySQL DB Product Microservice With 4 node cluster LoadBalancer CircuitBreaker UILayer WebServices BusinessLogic DatabaseLayer Customer Redis DB Customer Microservice With 2 node cluster â In this model Developers write the code in every Microservice to download the required configuration from a Central server (Ex. Spring Config Server for the Java World). â This creates an explicit dependency order in which service to come up will be critical. Config Server 4 82. Software Network Stack Vs Network Stack 11/19/2019 82 Pattern Software Stack Java Software Stack .NET Kubernetes 1 API Gateway Zuul Server SteelToe Istio Envoy 2 Service Discovery Eureka Server SteelToe Kube DNS 3 Load Balancer Ribbon Server SteelToe Istio Envoy 4 Circuit Breaker Hysterix SteelToe Istio 5 Config Server Spring Config SteelToe Secrets, Env - K8s Master Web Site https://netflix.github.io/ https://steeltoe.io/ https://kubernetes.io/ The Developer needs to write code to integrate with the Software Stack (Programming Language Specific. For Ex. Every microservice needs to subscribe to Service Discovery when the Microservice boots up. Service Discovery in Kubernetes is based on the Labels assigned to Pod and Services and its Endpoints (IP Address) are dynamically mapped (DNS) based on the Label. 4 83. Let-it-Crash Design Pattern â Erlang Philosophy 11/19/2019 83 â The Erlang view of the world is that everything is a process and that processes can interact only by exchanging messages. â A typical Erlang program might have hundreds, thousands, or even millions of processes. â Letting processes crash is central to Erlang. Itâs the equivalent of unplugging your router and plugging it back in â as long as you can get back to a known state, this turns out to be a very good strategy. â To make that happen, you build supervision trees. â A supervisor will decide how to deal with a crashed process. It will restart the process, or possibly kill some other processes, or crash and let someone else deal with it. â Two models of concurrency: Shared State Concurrency, & Message Passing Concurrency. The programming world went one way (toward shared state). The Erlang community went the other way. â All languages such as C, Java, C++, and so on, have the notion that there is this stuff called state and that we can change it. The moment you share something you need to bring Mutex a Locking Mechanism. â Erlang has no mutable data structures (thatâs not quite true, but itâs true enough). No mutable data structures = No locks. No mutable data structures = Easy to parallelize. 4 84. Let-it-Crash Design Pattern 11/19/2019 84 1. The idea of Messages as the first class citizens of a system, has been rediscovered by the Event Sourcing / CQRS community, along with a strong focus on domain models. 2. Event Sourced Aggregates are a way to Model the Processes and NOT things. 3. Each component MUST tolerate a crash and restart at any point in time. 4. All interaction between the components must tolerate that peers can crash. This mean ubiquitous use of timeouts and Circuit Breaker. 5. Each component must be strongly encapsulated so that failures are fully contained and cannot spread. 6. All requests sent to a component MUST be self describing as is practical so that processing can resume with a little recovery cost as possible after a restart. 4 85. Let-it-Crash : Comparison Erlang Vs. Microservices Vs. Monolithic Apps 85 Erlang Philosophy Micro Services Architecture Monolithic Apps (Java, C++, C#, Node JS ...) 1 Perspective Everything is a Process Event Sourced Aggregates are a way to model the Process and NOT things. Things (defined as Objects) and Behaviors 2 Crash Recovery Supervisor will decide how to handle the crashed process Kubernetes Manager monitors all the Pods (Microservices) and its Readiness and Health. K8s terminates the Pod if the health is bad and spawns a new Pod. Circuit Breaker Pattern is used handle the fallback mechanism. Not available. Most of the monolithic Apps are Stateful and Crash Recovery needs to be handled manually and all languages other than Erlang focuses on defensive programming. 3 Concurrency Message Passing Concurrency Domain Events for state changes within a Bounded Context & Integration Events for external Systems. Mostly Shared State Concurrency 4 State Stateless : Mostly Immutable Structures Immutability is handled thru Event Sourcing along with Domain Events and Integration Events. Predominantly Stateful with Mutable structures and Mutex as a Locking Mechanism 5 Citizen Messages Messages are 1st class citizen by Event Sourcing / CQRS pattern with a strong focus on Domain Models Mutable Objects and Strong focus on Domain Models and synchronous communication. 4 86. Day 1 - Summary 86 Setup 1. Setting up Kubernetes Cluster â 1 Master and â 2 Worker nodes Getting Started 1. Create Pods 2. Create ReplicaSets 3. Create Deployments 4. Rollouts and Rollbacks 5. Create Service 6. Create Ingress 7. App Auto Scaling App Setup 1. Secrets 2. Environments 3. ConfigMap 4. PodPresets 4 On Premise Setup 1. Setting up External Load Balancer using Metal LB 2. Setting up nginx Ingress Controller Infrastructure Design Patterns 1. API Gateway 2. Service Discovery 3. Load Balancer 4. Config Server 5. Circuit Breaker 6. Service Aggregator Pattern 7. Let It Crash Pattern Running Shopping Portal App 1. UI 2. Product Service 3. Product Review Service 4. MySQL Database 87. 19-11-2019 875 K8s Packet Path â Kubernetes Networking â Compare Docker and Kubernetes Networking â Pod to Pod Networking within the same Node â Pod to Pod Networking across the Node â Pod to Service Networking â Ingress - Internet to Service Networking â Egress â Pod to Internet Networking 88. Kubernetes Networking Mandatory requirements for Network implementation 1. All Pods can communicate with All other Pods without using Network Address Translation (NAT). 2. All Nodes can communicate with all the Pods without NAT. 3. The IP that is assigned to a Pod is the same IP the Pod sees itself as well as all other Pods in the cluster.19-11-2019 88 Source: https://github.com/meta-magic/kubernetes_workshop 5 89. 89 Container 1 172.17.3.2 Web Server 8080 Veth: eth0 Container 2 172.17.3.3 Microservice 9002 Veth: eth0 Container 3 172.17.3.4 Microservice 9003 Veth: eth0 Container 4 172.17.3.5 Microservice 9004 Veth: eth0 IP tables rules eth0 10.130.1.101/24 Node 1 Docker0 Bridge 172.17.3.1/16 Veth0 Veth1 Veth2 Veth3 Container 1 172.17.3.2 Web Server 8080 Veth: eth0 Container 2 172.17.3.3 Microservice 9002 Veth: eth0 Container 3 172.17.3.4 Microservice 9003 Veth: eth0 Container 4 172.17.3.5 Microservice 9004 Veth: eth0 IP tables rules eth0 10.130.1.102/24 Node 2 Docker0 Bridge 172.17.3.1/16 Veth0 Veth1 Veth2 Veth3 Docker Networking Vs. Kubernetes Networking Pod 1 172.17.3.2 Web Server 8080 Veth: eth0 Pod 2 172.17.3.3 Microservice 9002 Veth: eth0 Pod 3 172.17.3.4 Microservice 9003 Veth: eth0 Pod 4 172.17.3.5 Microservice 9004 Veth: eth0 IP tables rules eth0 10.130.1.101/24 Node 1 L2 Bridge 172.17.3.1/16 Veth0 Veth1 Veth2 Veth3 Same IP Range. NAT Required Uniq IP Range. netFilter, IP Tables / IPVS. No NAT required 5 Pod 1 172.17.3.6 Web Server 8080 Veth: eth0 Pod 2 172.17.3.7 Microservice 9002 Veth: eth0 Pod 3 172.17.3.8 Microservice 9003 Veth: eth0 Pod 4 172.17.3.9 Microservice 9004 Veth: eth0 IP tables rules eth0 10.130.1.102/24 Node 2 L2 Bridge 172.17.3.1/16 Veth0 Veth1 Veth2 Veth3 90. Kubernetes Networking 3 Networks Networks 1. Physical Network 2. Pod Network 3. Service Network 19-11-2019 90 Source: https://github.com/meta-magic/kubernetes_workshop CIDR Range (RFC 1918) 1. 10.0.0.0/8 2. 172.0.0.0/11 3. 192.168.0.0/16 Keep the Address ranges separate â Best Practices RFC 1918 1. Class A 2. Class B 3. Class C 5 91. Kubernetes Networking 3 Networks 91 Source: https://github.com/meta-magic/kubernetes_workshop eth0 10.130.1.102/24 Node 1 veth0 eth0 Pod 1 Container 1 172.17.4.1 eth0 Pod 2 Container 1 172.17.4.2 veth1 eth0 10.130.1.103/24 Node 2 veth1 eth0 Pod 1 Container 1 172.17.5.1 eth0 10.130.1.104/24 Node 3 veth1 eth0 Pod 1 Container 1 172.17.6.1 Service EP EP EP VIP 192.168.1.2/16 1. Physical Network 2. Pod Network 3. Service Network End Points handles dynamic IP Addresses of the Pods selected by a Service based on Pod Labels 5 Virtual IP doesnât have any physical network card or system attached. 92. Kubernetes: Pod to Pod Networking inside a Node By Default Linux has a Single Namespace and all the process in the namespace share the Network Stack. If you create a new namespace then all the process running in that namespace will have its own Network Stack, Routes, Firewall Rules etc. $ ip netns add namespace1 A mount point for namespace1 is created under /var/run/netns Create Namespace $ ip netns List Namespace eth0 10.130.1.101/24 Node 1 Root NW Namespace L2 Bridge 10.17.3.1/16 veth0 veth1 ForwardingTables BridgeimplementsARPtodiscoverlink- layerMACAddress eth0 Container 1 10.17.3.2 Pod 1 Container 2 10.17.3.2 eth0 Pod 2 Container 1 10.17.3.3 1. Pod 1 sends packet to eth0 â eth0 is connected to veth0 2. Bridge resolves the Destination with ARP protocol and 3. Bridge sends the packet to veth1 4. veth1 forwards the packet directly to Pod 2 thru eth0 1 2 4 3 This entire communication happens in localhost. So Data transfer speed will NOT be affected by Ethernet card speed. Kube Proxy 19-11-2019 925 93. eth0 10.130.1.102/24 Node 2 Root NW Namespace L2 Bridge 10.17.4.1/16 veth0 Kubernetes: Pod to Pod Networking Across Node eth0 10.130.1.101/24 Node 1 Root NW Namespace L2 Bridge 10.17.3.1/16 veth0 veth1 ForwardingTables eth0 Container 1 10.17.3.2 Pod 1 Container 2 10.17.3.2 eth0 Pod 2 Container 1 10.17.3.3 1. Pod 1 sends packet to eth0 â eth0 is connected to veth0 2. Bridge will try to resolve the Destination with ARP protocol and ARP will fail because there is no device connected to that IP. 3. On Failure Bridge will send the packet to eth0 of the Node 1. 4. At this point packet leaves eth0 and enters the Network and network routes the packet to Node 2. 5. Packet enters the Root namespace and routed to the L2 Bridge. 6. veth0 forwards the packet to eth0 of Pod 3 1 2 4 3 eth0 Pod 3 Container 1 10.17.4.1 5 6 Kube ProxyKube Proxy Src-IP:Port: Pod1:17711 â Dst-IP:Port: Pod3:80 19-11-2019 935 94. eth0 10.130.1.102/24 Node 2 Root NW Namespace L2 Bridge 10.17.4.1/16 veth0 Kubernetes: Pod to Service to Pod â Load Balancer eth0 10.130.1.101/24 Node 1 Root NW Namespace L2 Bridge 10.17.3.1/16 veth0 veth1 ForwardingTables eth0 Container 1 10.17.3.2 Pod 1 Container 2 10.17.3.2 eth0 Pod 2 Container 1 10.17.3.3 1. Pod 1 sends packet to eth0 â eth0 is connected to veth0 2. Bridge will try to resolve the Destination with ARP protocol and ARP will fail because there is no device connected to that IP. 3. On Failure Bridge will give the packet to Kube Proxy 4. it goes thru ip tables rules installed by Kube Proxy and rewrites the Dst-IP with Pod3-IP. IPVS has done the Cluster load Balancing directly on the node and packet is given to eth0 of the Node1. 5. Now packet leaves Node 1 eth0 and enters the Network and network routes the packet to Node 2. 6. Packet enters the Root namespace and routed to the L2 Bridge. 7. veth0 forwards the packet to eth0 of Pod 3 1 2 4 3 eth0 Pod 3 Container 1 10.17.4.1 5 6 Kube ProxyKube Proxy 7 SrcIP:Port: Pod1:17711 â Dst-IP:Port: Service1:80 Src-IP:Port: Pod1:17711 â Dst-IP:Port: Pod3:80 19-11-2019 945 95. eth0 10.130.1.102/24 Node 2 Root NW Namespace L2 Bridge 10.17.4.1/16 veth0 Kubernetes Pod to Service to Pod â Return Journey eth0 10.130.1.101/24 Node 1 Root NW Namespace L2 Bridge 10.17.3.1/16 veth0 veth1 ForwardingTables eth0 Container 1 10.17.3.2 Pod 1 Container 2 10.17.3.2 eth0 Pod 2 Container 1 10.17.3.3 1. Pod 3 receives data from Pod 1 and sends the reply back with Source as Pod3 and Destination as Pod1 2. Bridge will try to resolve the Destination with ARP protocol and ARP will fail because there is no device connected to that IP. 3. On Failure Bridge will give the packet Node 2 eth0 4. Now packet leaves Node 2 eth0 and enters the Network and network routes the packet to Node 1. (Dst = Pod1) 5. it goes thru ip tables rules installed by Kube Proxy and rewrites the Src-IP with Service-IP. Kube Proxy gives the packet to L2 Bridge. 6. L2 bridge makes the ARP call and hand over the packet to veth0 7. veth0 forwards the packet to eth0 of Pod1 1 2 4 3 eth0 Pod 3 Container 1 10.17.4.1 5 6 Kube ProxyKube Proxy 7 Src-IP: Pod3:80 â Dst-IP:Port: Pod1:17711Src-IP:Port: Service1:80â Dst-IP:Port: Pod1:17711 19-11-2019 955 96. eth0 10.130.1.102/24 Node X Root NW Namespace L2 Bridge 10.17.4.1/16 veth0 Kubernetes: Internet to Pod 1. Client Connects to App published Domain. 2. Once the Ingress Load Balancer receives the packet it picks a VM (K8s Node). 3. Once inside the VM IP Tables knows how to redirect the packet to the Pod using internal load Balancing rules installed into the cluster using Kube Proxy. 4. Traffic enters Kubernetes cluster and reaches the Node X (10.130.1.102). 5. Node X gives the packet to the L2 Bridge 6. L2 bridge makes the ARP call and hand over the packet to veth0 7. veth0 forwards the packet to eth0 of Pod 8 1 2 4 3 5 6 7 Src: Client IP â Dst: App Dst Src: Client IP â Dst: Pod IP Ingress Load Balancer Client / User Src: Client IP â Dst: VM-IP eth0 Pod 8 Container 1 10.17.4.1 Kube Proxy 19-11-2019 96 VM VMVM 5 97. Kubernetes: Pod to Internet eth0 10.130.1.101/24 Node 1 Root NW Namespace L2 Bridge 10.17.3.1/16 veth0 veth1 ForwardingTables eth0 Container 1 10.17.3.2 Pod 1 Container 2 10.17.3.2 eth0 Pod 2 Container 1 10.17.3.3 1. Pod 1 sends packet to eth0 â eth0 is connected to veth0 2. Bridge will try to resolve the Destination with ARP protocol and ARP will fail because there is no device connected to that IP. 3. On Failure Bridge will give the packet to IP Tables 4. The Gateway will reject the Pod IP as it will recognize only the VM IP. So source IP is replaced with VM-IP 5. Packet enters the network and routed to Internet Gateway. 6. Packet reaches the GW and it replaces the VM-IP (internal) with an External IP. 7. Packet Reaches External Site (Google) 1 2 4 3 5 6 Kube Proxy 7 Src: Pod1 â Dst: Google Src: VM-IP â Dst: Google Gateway Google Src: Ex-IP â Dst: Google On the way back the packet follows the same path and any Src IP mangling is un done and each layer understands VM-IP and Pod IP within Pod Namespace. 97 VM 5 98. 19-11-2019 98 Kubernetes Networking Advanced â Kubernetes IP Network â OSI Layer | L2 | L3 | L4 | L7 | â IP Tables | IPVS | BGP | VXLAN â Kubernetes DNS â Kubernetes Proxy â Kubernetes Load Balancer, Cluster IP, Node Port â Kubernetes Ingress â Kubernetes Ingress â Amazon Load Balancer â Kubernetes Ingress â Metal LB (On Premise) 6 99. Kubernetes Network Requirements 19-11-2019 99 Source: https://github.com/meta-magic/kubernetes_workshop 1. IPAM (IP Address Management & Life cycle Management of Network Devices 2. Connectivity and Container Network 3. Route Advertisement 6 100. 19-11-2019 100 OSI Layers 6 101. Networking Glossary Netfilter â Packet Filtering in Linux Software that does packet filtering, NAT and other Packet mangling IP Tables It allows Admin to configure the netfilter for managing IP traffic. ConnTrack Conntrack is built on top of netfilter to handle connection tracking.. IPVS â IP Virtual Server Implements a transport layer load balancing as part of the Linux Kernel. Itâs similar to IP Tables and based on netfilter hook function and uses hash table for the lookup. 101 Border Gateway Protocol BGP is a standardized exterior gateway protocol designed to exchange routing and reachability information among autonomous systems (AS) on the Internet. The protocol is often classified as a path vector protocol but is sometimes also classed as a distance-vector routing protocol. Some of the well known & mandatory attributes are AS Path, Next Hop Origin. 6 L2 Bridge (Software Switch) Network devices, called switches (or bridges) are responsible for connecting several network links to each other, creating a LAN. Major components of a network switch are a set of network ports, a control plane, a forwarding plane, and a MAC learning database. The set of ports are used to forward traffic between other switches and end-hosts in the network. The control plane of a switch is typically used to run the Spanning Tree Protocol, that calculates a minimum spanning tree for the LAN, preventing physical loops from crashing the network. The forwarding plane is responsible for processing input frames from the network ports and making a forwarding decision on which network port or ports the input frame is forwarded to. 102. Networking Glossary Layer 2 Networking Layer 2 is the Data Link Layer (OSI Mode) providing Node to Node Data Transfer. Layer 2 deals with delivery of frames between 2 adjacent nodes on a network. Ethernet is an Ex. Of Layer 2 networking with MAC represented as a Sub Layer. Flannel uses L3 with VXLAN (L2) networking. Layer 4 Networking Transport layer controls the reliability of a given link through flow control. Layer 7 Networking Application layer networking (HTTP, FTP etc.,) This is the closet layer to the end user. Kubernetes Ingress Controller is a L7 Load Balancer. 102 Layer 3 Networking Layer 3âs primary concern involves routing packets between hosts on top of the layer 2 connections. IPv4, IPv6, and ICMP are examples of Layer 3 networking protocols. Calico uses L3 networking. VXLAN Networking Virtual Extensible LAN used to help large cloud deployments by encapsulating L2 Frames within UDP Datagrams. VXLAN is similar to VLAN (which has a limitation of 4K network IDs). VXLAN is an encapsulation and overlay protocol that runs on top of existing Underlay networks. VXLAN can have 16 million Network IIDs. Overlay Networking An overlay network is a virtual, logical network built on top of an existing network. Overlay networks are often used to provide useful abstractions on top of existing networks and to separate and secure different logical networks. 6 Source Network Address Translation SNAT refers to a NAT procedure that modifies the source address of an IP Packet. Destination Network Address Translation DNAT refers to a NAT procedure that modifies the Destination address of an IP Packet. 103. eth0 10.130.1.102 Node / Server 1 172.17.4.1 VSWITCH 172.17.4.1 Customer 1 Customer 2 eth0 10.130.2.187 Node / Server 2 172.17.5.1 VSWITCH 172.17.5.1 Customer 1 Customer 2 VXLAN Encapsulation 1036 10.130.1.0/24 10.130.2.0/24 Underlay Network VSWITCH: Virtual Switch Switch SwitchRouter 104. eth0 10.130.1.102 Node / Server 1 172.17.4.1 VSWITCH VTEP 172.17.4.1 Customer 1 Customer 2 eth0 10.130.2.187 Node / Server 2 172.17.5.1 VSWITCH VTEP 172.17.5.1 Customer 1 Customer 2 VXLAN Encapsulation 1046 Overlay Network VSWITCH: Virtual Switch. | VTEP : Virtual Tunnel End Point VXLAN encapsulate L2 into UDP packets tunneling using L3. This means no specialized hardware required. So, the Overlay networks could be created purely in Software. VLAN = 4094 (2 reserved) Networks VNI = 16 Million Networks (24-bit ID) 105. eth0 10.130.1.102 Node / Server 1 172.17.4.1 VSWITCH VTEP 172.17.4.1 Customer 1 Customer 2 eth0 10.130.2.187 Node / Server 2 172.17.5.1 VSWITCH VTEP 172.17.5.1 Customer 1 Customer 2 VXLAN Encapsulation 1056 Overlay Network ARP Broadcast ARP BroadcastARP Broadcast Multicast VSWITCH: Virtual Switch. | VTEP : Virtual Tunnel End Point ARP Unicast 106. eth0 10.130.1.102 Node / Server 1 172.17.4.1 B1 â MAC VSWITCH VTEP 172.17.4.1 Y1 â MAC Customer 1 Customer 2 eth0 10.130.2.187 Node / Server 2 172.17.5.1 B2 â MAC VSWITCH VTEP 172.17.5.1 Y2 â MAC Customer 1 Customer 2 VXLAN Encapsulation 1066 Overlay Network Src: 172.17.4.1 Src: B1 â MAC Dst: 172.17.5.1 Dst: B2 - MAC Src: 10.130.1.102 Dst: 10.130.2.187 Src UDP Port: Dynamic Dst UDP Port: 4789 VNI: 100 Src: 172.17.4.1 Src: B1 â MAC Dst: 172.17.5.1 Dst: B2 - MAC Src: 172.17.4.1 Src: B1 â MAC Dst: 172.17.5.1 Dst: B2 - MAC VSWITCH: Virtual Switch. | VTEP : Virtual Tunnel End Point | VNI : Virtual Network Identifier 107. eth0 10.130.1.102 Node / Server 1 172.17.4.1 B1 â MAC VSWITCH VTEP 172.17.4.1 Y1 â MAC Customer 1 Customer 2 eth0 10.130.2.187 Node / Server 2 172.17.5.1 B2 â MAC VSWITCH VTEP 172.17.5.1 Y2 â MAC Customer 1 Customer 2 VXLAN Encapsulation 1076 Overlay Network Src: 10.130.2.187 Dst: 10.130.1.102 Src UDP Port: Dynamic Dst UDP Port: 4789 VNI: 100 VSWITCH: Virtual Switch. | VTEP : Virtual Tunnel End Point | VNI : Virtual Network Identifier Src: 172.17.5.1 Src: B2 - MAC Dst: 172.17.4.1 Dst: B1 â MAC Src: 172.17.5.1 Src: B2 - MAC Dst: 172.17.4.1 Dst: B1 â MAC Src: 172.17 https://www.friendbookmark.com/videos/968/modern-microservices-architecture-with-dockerMicroservices are all the rage these days. Docker is a tool which makes managing Microservices a whole lot easier. But what do Microservices really mean? What are the best practices of composing your application with Microservices? How can you leverage Docker and the public cloud to help you build a more agile DevOps process? How does the Azure Container Service fit in? Join us to find out the answer. Topics covered in this presentation slides: 1. Modern Microservices Architecture with Docker Alon Fliess Chief Software Architect Microsoft Regional Director alonf@codevalue.net http://alonfliess.me 1 Eran Stiller Cloud Division Leader Senior Software Architect erans@codevalue.net http://stiller.co.il/blog 2. Agenda Software Architecture Micro Services Architecture Containers & Docker Azure Container Services Demo 2 3. About Alon Chief Architect & Co-Founder at CodeValue More than 25 years of hands-on experience Microsoft Regional Director & Microsoft MVP Active member of Microsoft Advisory Councils Renowned speaker at both international and domestic events 3 4. About Eran Cloud Division Leader & Co-Founder at CodeValue Software architect, consultant and instructor Microsoft Azure MVP More than 10 years of hands-on experience Expert in large-scale, server-side, highly-concurrent systems Co-Founder of Azure Israel Meetup 4 5. About CodeValue & OzCode CodeValue â Inspiring Code. Creating Value. A consultancy and software development firm High quality software projects and managed services Valuable training and mentoring Advanced software technology and methodology consulting OzCode â an innovative VS debugging extension Helps identify and fix bugs during C# development Saves time and effort Makes the debugging process easy and simple 6. 6 Software Architecture 7. What a Modern Architect SHOULD Know 7 SRS UML Use Cases User Stories Risk Mitigations Design Redis SRP 7 Client Server Schema OOP SQL SOA NoSQL Encapsulation Data JSON ETL Reporting AWS BASE Queue XML Cloud Big Data ACID Micro Services Architecture Distributed Cache REST Idempotency BI Map Reduce NLB HTTP Indexing CDN Search Security Log HPC Authentication Docker Containers HA DRPub/Sub UI/UX CQRS MEAN ORM Push Notifications 8. What a Modern Architect MUST Know Understand the requirements Understand the constraints Low coupling High cohesion Balance of size & number of components Volatility Workplace politics 9. Software Architecture The system structure Built from software components The relationship between components The properties of both components and relations Software architecture is about making fundamental structural choices Which are costly to change once implemented Designing software architecture is a mix of art and science! 10. Software Architecture Goals Defining a structured solution that Meets all the technical and operational requirements Captures the non-volatile (hard to change) decisions Focuses on important elements of the system (context) Provide a solid foundation for a successful software product Answer the requirements: Functional Non-Functional (quality attributes) Constraints 10 11. Cohesion The degree to which a module performs one and only one function Strive for high cohesion A module can be: Library (assembly, shared module, DLL) File Class Method COM/CORBA component (Micro) Service Any reusable element 11 12. Coupling The degree to which each program module relies on each of the other modules Low coupling often correlates with high cohesion, and vice versa Low coupling is A sign of a well-structured computer system Good design When combined with high cohesion Supports high readability, maintainability, extendibility, and reusability Micro Service Architecture == High Cohesion & Low Coupling 12 13. Why Should I Care About Coupling Tightly coupled systems tend to exhibit the following developmental characteristics A change in one module usually forces a ripple (cascading) effect of changes in other modules Assembly of modules might require more effort and/or time due to the increased inter-module dependency A particular module might be harder to reuse and/or test because dependent modules must be included The DevOps process becomes a nightmare!!! 13 14. Fan-In Fan-Out and Stable Module One way to examine module stability (i.e. low coupling and high cohesion) is by looking at its fan-in fan-out and other dependencies Fan-In The number of users of the module Fan-Out The number of modules that the current module is being used by A stable module is a module that has high fun-in and low fan-out This module can be easily reused 14 15. Just The Right Number Of Modules 15 number of modules Module integration cost module development cost The cost of the software 16. Modern Software Project Challenges 16 We need to do more with less More functional requirements and better quality attributes (Many) more end users (Many) more servers to manage with less operators More changes with less or even no maintenance downtime 17. The 24/7 Challenge How do you update a system running 24/7/365? How do you keep the application servers responsive? How do you keep all application servers synced? How do you update the data/schema? How do you update all your clientsâ software? Web, Mobile, Desktopâ How do you rollback on error? How do you rollback data? How do you know there is an error? 17 18. The 24/7 Challenge Plan ahead DevOps Architect Low Coupling, High Cohesion Schema/API Versioning Use supporting platforms Cloud Containers 18 19. Microservices Architecture (MSA) - Wikipedia âMicroservices is a specialization of and implementation approach for service-oriented architectures (SOA) used to build flexible, independently deployable software systemsâ âServices are small in size, messaging enabled, bounded by contexts, autonomously developed, independently deployable, decentralized and built and released with automated processesâ âThe benefit of distributing different responsibilities of the system into different smaller services is that it enhances the cohesion and decreases the couplingâ 19 20. MSA Criticism The architecture introduces additional complexity such as: Network latency Message formats and versioning nightmare Load balancing and fault tolerance management Testing and deployment are more complicated 20 21. MSA Drawbacks Solution Plan ahead DevOps Architect Low Coupling, High Cohesion Schema/API Versioning Use supporting platforms Cloud Containers 21 22. 22 Modern Microservices Architecture with Docker 23. Docker, Docker, Docker Containers have been around for many years Docker Inc. did not invent them They created open source software to build and manage containers Docker makes containers easy Even I can create and run them Docker is a container format and a set of tools Docker CLI, Docker Engine, Docker Swarm, Docker Compose, Docker Machine and moreâ Docker is an eco-system 23 24. Server Host OS Hypervisor Server Host OS Docker Engine Guest OS Guest OS Guest OS Bins/Libs Bins/Libs Bins/Libs App A App Aâ App B Bins/Libs Bins/Libs AppA AppAâ AppB AppBâ AppB AppBâ AppB AppBâ Containers are isolated, but share OS and, where appropriate, bins/libraries Bins/Libs 25. Developer Workstation Docker Host Container Container Container 26. Developer Workstation Docker Engine Container Container Container Container Container Container 27. Developer Workstation Container Container ContainerContainer Container Container Container Container Container 28. App1 App2 29. Demo Architecture 33 Web Portal RedisTwitter Reader Queue Twitter Twitter Analyzer Queue Aggregator Pub/Sub Microsoft Cognitive Services 30. Demo Architecture 34 RedisTwitter Reader Web Portal Load Balancer Queue Twitter Queue Aggregator Pub/Sub Microsoft Cognitive Services Twitter Analyzer 31. Demo 35 https://github.com/estiller/tweet-analyzer-demo 32. Summary Software Architecture Micro services Architecture Containers & Docker Azure Container Services Demo 39 33. 41 https://www.friendbookmark.com/videos/967/caching-in-docker-the-hardest-thing-in-computer-sciencePresentation about problems encountered while building Breeze - Development Environment for Apache Airflow. Docker is great for containerisation but when you are trying to make full use of it for caching and development, there are a number of problems you have to overcome. Topics covered in this presentation slides: 1. The hardest thing in computer science 2. Hard things 3. Docker Caching Dependency versions Install dependencies [ 20 minutes or so ] Only here copy all sources 4. Intended behaviour ● No change: docker is not rebuilt - LIGHTNING FAST!!!! ● Sources change/dependencies not: only sources are added - QUITE FAST !!! ● Dependencies change: dependencies installed, sources - LITTLE SLOWER !! 5. Actual behaviour same machine - local checkout ● Local docker registry ● Repeated build: 1:06m ● Only sources: 1:30m ● Dependencies: 11m ● Whole build: ~ 30m 6. CI case ● Always fresh machine ○ no code ○ no registry ● Git clone/checkout ● Build ● Wipeout 7. Docker registry to the rescue! Build cache: ● Docker build ● Docker push airflow/airflow:latest Use cache: ● Docker pull airflow/airflow:latest ● docker build --cache-from ariflow/airflow:latest 8. Actual behaviour Docker Hub automated build ● DockerHub docker registry as cache ● Repeated build: 11m ● Only sources: 11m https://www.friendbookmark.com/videos/966/how-to-test-infrastructure-code-automated-testing-for-terraform-kubernetes-docker-packer-and-moreThis talk is a step-by-step, live-coding class on how to write automated tests for infrastructure code, including the code you write for use with tools such as Terraform, Kubernetes, Docker, and Packer. Topics covered include unit tests, integration tests, end-to-end tests, test parallelism, retries, error handling, static analysis, and more. Topics covered in this presentation slides: 1. Automated testing for: ✓ terraform ✓ docker ✓ packer ✓ kubernetes ✓ and more Passed: 5. Failed: 0. Skipped: 0. Test run successful. How to test infrastructure code 2. The DevOps world is full of Fear 3. Fear of outages 4. Fear of security breaches 5. Fear of data loss 6. Fear of change 7. âFear leads to anger. Anger leads to hate. Hate leads to suffering.â Scrum Master Yoda 8. And you all know what suffering leads to, right? 9. Credit: Daniele Polencic 10. Many DevOps teams deal with this fear in two ways: 11. 1) Heavy drinking and smoking 12. 2) Deploying less frequently 13. Sadly, both of these just make the problem worse! 14. Thereâs a better way to deal with this fear: 15. Automated tests 16. Automated tests give you the confidence to make changes 17. Fight fear with confidence 18. We know how to write automated tests for application codeâ 19. resource "aws_lambda_function" "web_app" { function_name = var.name role = aws_iam_role.lambda.arn # ... } resource "aws_api_gateway_integration" "proxy" { type = "AWS_PROXY" uri = aws_lambda_function.web_app.invoke_arn # ... } But how do you test your Terraform code deploys infrastructure that works? 20. apiVersion: apps/v1 kind: Deployment metadata: name: hello-world-app-deployment spec: selector: matchLabels: app: hello-world-app replicas: 1 spec: containers: - name: hello-world-app image: gruntwork-io/hello-world-app:v1 ports: - containerPort: 8080 How do you test your Kubernetes code configures your services correctly? 21. This talk is about how to write tests for your infrastructure code. 22. Iâm Yevgeniy Brikman ybrikman.com 23. Co-founder of Gruntwork gruntwork.io 24. Author 25. 1. Static analysis 2. Unit tests 3. Integration tests 4. End-to-end tests 5. Conclusion Outline 26. 1. Static analysis 2. Unit tests 3. Integration tests 4. End-to-end tests 5. Conclusion Outline 27. Static analysis: test your code without deploying it. 28. Static analysis 1. Compiler / parser / interpreter 2. Linter 3. Dry run 29. Static analysis 1. Compiler / parser / interpreter 2. Linter 3. Dry run 30. Statically check your code for syntactic and structural issues 31. Tool Command Terraform terraform validate Packer packer validate Kubernetes kubectl apply -f --dry-run --validate=true Examples: 32. Static analysis 1. Compiler / parser / interpreter 2. Linter 3. Dry run 33. Statically validate your code to catch common errors 34. Tool Linters Terraform 1. conftest 2. terraform_validate 3. tflint Docker 1. dockerfile_lint 2. hadolint 3. dockerfilelint Kubernetes 1. kube-score 2. kube-lint 3. yamllint Examples: 35. Static analysis 1. Compiler / parser / interpreter 2. Linter 3. Dry run 36. Partially execute the code and validate the âplanâ, but donât actually deploy 37. Tool Dry run options Terraform 1. terraform plan 2. HashiCorp Sentinel 3. terraform-compliance Kubernetes kubectl apply -f --server-dry-run Examples: 38. 1. Static analysis 2. Unit tests 3. Integration tests 4. End-to-end tests 5. Conclusion Outline 39. Unit tests: test a single âunitâ works in isolation. 40. Unit tests 1. Unit testing basics 2. Example: Terraform unit tests 3. Example: Docker/Kubernetes unit tests 4. Cleaning up after tests 41. Unit tests 1. Unit testing basics 2. Example: Terraform unit tests 3. Example: Docker/Kubernetes unit tests 4. Cleaning up after tests 42. You canât âunit testâ an entire end- to-end architecture 43. Instead, break your infra code into small modules and unit test those! module module module module module module module module module module module module module module module 44. With app code, you can test units in isolation from the outside world 45. resource "aws_lambda_function" "web_app" { function_name = var.name role = aws_iam_role.lambda.arn # ... } resource "aws_api_gateway_integration" "proxy" { type = "AWS_PROXY" uri = aws_lambda_function.web_app.invoke_arn # ... } But 99% of infrastructure code is about talking to the outside worldâ 46. resource "aws_lambda_function" "web_app" { function_name = var.name role = aws_iam_role.lambda.arn # ... } resource "aws_api_gateway_integration" "proxy" { type = "AWS_PROXY" uri = aws_lambda_function.web_app.invoke_arn # ... } If you try to isolate a unit from the outside world, youâre left with nothing! 47. So you can only test infra code by deploying to a real environment 48. Key takeaway: thereâs no pure unit testing for infrastructure code. 49. Therefore, the test strategy is: 1. Deploy real infrastructure 2. Validate it works (e.g., via HTTP requests, API calls, SSH commands, etc.) 3. Undeploy the infrastructure (So itâs really integration testing of a single unit!) 50. Tool Deploy / Undeploy Validate Works with Terratest Yes Yes Terraform, Kubernetes, Packer, Docker, Servers, Cloud APIs, etc. kitchen-terraform Yes Yes Terraform Inspec No Yes Servers, Cloud APIs Serverspec No Yes Servers Goss No Yes Servers Tools that help with this strategy: 51. Tool Deploy / Undeploy Validate Works with Terratest Yes Yes Terraform, Kubernetes, Packer, Docker, Servers, Cloud APIs, etc. kitchen-terraform Yes Yes Terraform Inspec No Yes Servers, Cloud APIs Serverspec No Yes Servers Goss No Yes Servers In this talk, weâll use Terratest: 52. Unit tests 1. Unit testing basics 2. Example: Terraform unit tests 3. Example: Docker/Kubernetes unit tests 4. Cleaning up after tests 53. Sample code for this talk is at: github.com/gruntwork-io/infrastructure-as-code-testing-talk 54. An example of a Terraform module you may want to test: 55. infrastructure-as-code-testing-talk └ examples └ hello-world-app └ main.tf └ outputs.tf └ variables.tf └ modules └ test └ README.md hello-world-app: deploy a âHello, Worldâ web service 56. resource "aws_lambda_function" "web_app" { function_name = var.name role = aws_iam_role.lambda.arn # ... } resource "aws_api_gateway_integration" "proxy" { type = "AWS_PROXY" uri = aws_lambda_function.web_app.invoke_arn # ... } Under the hood, this example runs on top of AWS Lambda & API Gateway 57. $ terraform apply Outputs: url = ruvvwv3sh1.execute-api.us-east-2.amazonaws.com $ curl ruvvwv3sh1.execute-api.us-east-2.amazonaws.com Hello, World! When you run terraform apply, it deploys and outputs the URL 58. Letâs write a unit test for hello-world-app with Terratest 59. infrastructure-as-code-testing-talk └ examples └ modules └ test └ hello_world_app_test.go └ README.md Create hello_world_app_test.go 60. func TestHelloWorldAppUnit(t *testing.T) { terraformOptions := &terraform.Options{ TerraformDir: "../examples/hello-world-app", } defer terraform.Destroy(t, terraformOptions) terraform.InitAndApply(t, terraformOptions) validate(t, terraformOptions) } The basic test structure 61. func TestHelloWorldAppUnit(t *testing.T) { terraformOptions := &terraform.Options{ TerraformDir: "../examples/hello-world-app", } defer terraform.Destroy(t, terraformOptions) terraform.InitAndApply(t, terraformOptions) validate(t, terraformOptions) } 1. Tell Terratest where your Terraform code lives 62. func TestHelloWorldAppUnit(t *testing.T) { terraformOptions := &terraform.Options{ TerraformDir: "../examples/hello-world-app", } defer terraform.Destroy(t, terraformOptions) terraform.InitAndApply(t, terraformOptions) validate(t, terraformOptions) } 2. Run terraform init and terraform apply to deploy your module 63. func TestHelloWorldAppUnit(t *testing.T) { terraformOptions := &terraform.Options{ TerraformDir: "../examples/hello-world-app", } defer terraform.Destroy(t, terraformOptions) terraform.InitAndApply(t, terraformOptions) validate(t, terraformOptions) } 3. Validate the infrastructure works. Weâll come back to this shortly. 64. func TestHelloWorldAppUnit(t *testing.T) { terraformOptions := &terraform.Options{ TerraformDir: "../examples/hello-world-app", } defer terraform.Destroy(t, terraformOptions) terraform.InitAndApply(t, terraformOptions) validate(t, terraformOptions) } 4. Run terraform destroy at the end of the test to undeploy everything 65. func validate(t *testing.T, opts *terraform.Options) { url := terraform.Output(t, opts, "url") http_helper.HttpGetWithRetry(t, url, // URL to test 200, // Expected status code "Hello, World!", // Expected body 10, // Max retries 3 * time.Second // Time between retries ) } The validate function 66. func validate(t *testing.T, opts *terraform.Options) { url := terraform.Output(t, opts, "url") http_helper.HttpGetWithRetry(t, url, // URL to test 200, // Expected status code "Hello, World!", // Expected body 10, // Max retries 3 * time.Second // Time between retries ) } 1. Run terraform output to get the web service URL 67. func validate(t *testing.T, opts *terraform.Options) { url := terraform.Output(t, opts, "url") http_helper.HttpGetWithRetry(t, url, // URL to test 200, // Expected status code "Hello, World!", // Expected body 10, // Max retries 3 * time.Second // Time between retries ) } 2. Make HTTP requests to the URL 68. func validate(t *testing.T, opts *terraform.Options) { url := terraform.Output(t, opts, "url") http_helper.HttpGetWithRetry(t, url, // URL to test 200, // Expected status code "Hello, World!", // Expected body 10, // Max retries 3 * time.Second // Time between retries ) } 3. Check the response for an expected status and body 69. func validate(t *testing.T, opts *terraform.Options) { url := terraform.Output(t, opts, "url") http_helper.HttpGetWithRetry(t, url, // URL to test 200, // Expected status code "Hello, World!", // Expected body 10, // Max retries 3 * time.Second // Time between retries ) } 4. Retry the request up to 10 times, as deployment is asynchronous 70. Note: since weâre testing a web service, we use HTTP requests to validate it. 71. Infrastructure Example Validate withâ Example Web service Dockerized web app HTTP requests Terratest http_helper package Server EC2 instance SSH commands Terratest ssh package Cloud service SQS Cloud APIs Terratest aws or gcp packages Database MySQL SQL queries MySQL driver for Go Examples of other ways to validate: 72. $ export AWS_ACCESS_KEY_ID=xxxx $ export AWS_SECRET_ACCESS_KEY=xxxxx To run the test, first authenticate to AWS 73. $ go test -v -timeout 15m -run TestHelloWorldAppUnit â --- PASS: TestHelloWorldAppUnit (31.57s) Then run go test. You now have a unit test you can run after every commit! 74. Unit tests 1. Unit testing basics 2. Example: Terraform unit tests 3. Example: Docker/Kubernetes unit tests 4. Cleaning up after tests 75. What about other tools, such as Docker + Kubernetes? 76. infrastructure-as-code-testing-talk └ examples └ hello-world-app └ docker-kubernetes └ Dockerfile └ deployment.yml └ modules └ test └ README.md docker-kubernetes: deploy a âHello, Worldâ web service to Kubernetes 77. FROM ubuntu:18.04 EXPOSE 8080 RUN DEBIAN_FRONTEND=noninteractive apt-get update && apt-get install -y busybox RUN echo 'Hello, World!' > index.html CMD ["busybox", "httpd", "-f", "-p", "8080"] Dockerfile: Dockerize a simple âHello, World!â web service 78. apiVersion: apps/v1 kind: Deployment metadata: name: hello-world-app-deployment spec: selector: matchLabels: app: hello-world-app replicas: 1 spec: containers: - name: hello-world-app image: gruntwork-io/hello-world-app:v1 ports: - containerPort: 8080 deployment.yml: define how to deploy a Docker container in Kubernetes 79. $ cd examples/docker-kubernetes $ docker build -t gruntwork-io/hello-world-app:v1 . Successfully tagged gruntwork-io/hello-world-app:v1 $ kubectl apply -f deployment.yml deployment.apps/hello-world-app-deployment created service/hello-world-app-service created $ curl localhost:8080 Hello, World! Build the Docker image, deploy to Kubernetes, and check URL 80. Letâs write a unit test for this code. 81. infrastructure-as-code-testing-talk └ examples └ modules └ test └ hello_world_app_test.go └ docker_kubernetes_test.go └ README.md Create docker_kubernetes_test.go 82. func TestDockerKubernetes(t *testing.T) { buildDockerImage(t) path := "../examples/docker-kubernetes/deployment.yml" options := k8s.NewKubectlOptions("", "", "") defer k8s.KubectlDelete(t, options, path) k8s.KubectlApply(t, options, path) validate(t, options) } The basic test structure 83. func TestDockerKubernetes(t *testing.T) { buildDockerImage(t) path := "../examples/docker-kubernetes/deployment.yml" options := k8s.NewKubectlOptions("", "", "") defer k8s.KubectlDelete(t, options, path) k8s.KubectlApply(t, options, path) validate(t, options) } 1. Build the Docker image. Youâll see the buildDockerImage method shortly. 84. func TestDockerKubernetes(t *testing.T) { buildDockerImage(t) path := "../examples/docker-kubernetes/deployment.yml" options := k8s.NewKubectlOptions("", "", "") defer k8s.KubectlDelete(t, options, path) k8s.KubectlApply(t, options, path) validate(t, options) } 2. Tell Terratest where your Kubernetes deployment is defined 85. func TestDockerKubernetes(t *testing.T) { buildDockerImage(t) path := "../examples/docker-kubernetes/deployment.yml" options := k8s.NewKubectlOptions("", "", "") defer k8s.KubectlDelete(t, options, path) k8s.KubectlApply(t, options, path) validate(t, options) } 3. Configure kubectl options to authenticate to Kubernetes 86. func TestDockerKubernetes(t *testing.T) { buildDockerImage(t) path := "../examples/docker-kubernetes/deployment.yml" options := k8s.NewKubectlOptions("", "", "") defer k8s.KubectlDelete(t, options, path) k8s.KubectlApply(t, options, path) validate(t, options) } 4. Run kubectl apply to deploy the web app to Kubernetes 87. func TestDockerKubernetes(t *testing.T) { buildDockerImage(t) path := "../examples/docker-kubernetes/deployment.yml" options := k8s.NewKubectlOptions("", "", "") defer k8s.KubectlDelete(t, options, path) k8s.KubectlApply(t, options, path) validate(t, options) } 5. Check the app is working. Youâll see the validate method shortly. 88. func TestDockerKubernetes(t *testing.T) { buildDockerImage(t) path := "../examples/docker-kubernetes/deployment.yml" options := k8s.NewKubectlOptions("", "", "") defer k8s.KubectlDelete(t, options, path) k8s.KubectlApply(t, options, path) validate(t, options) } 6. At the end of the test, remove all Kubernetes resources you deployed 89. func buildDockerImage(t *testing.T) { options := &docker.BuildOptions{ Tags: []string{"gruntwork-io/hello-world-app:v1"}, } path := "../examples/docker-kubernetes" docker.Build(t, path, options) } The buildDockerImage method 90. func validate(t *testing.T, opts *k8s.KubectlOptions) { k8s.WaitUntilServiceAvailable(t, opts, "hello-world- app-service", 10, 1*time.Second) http_helper.HttpGetWithRetry(t, serviceUrl(t, opts), // URL to test 200, // Expected status code "Hello, World!", // Expected body 10, // Max retries 3*time.Second // Time between retries ) } The validate method 91. func validate(t *testing.T, opts *k8s.KubectlOptions) { k8s.WaitUntilServiceAvailable(t, opts, "hello-world- app-service", 10, 1*time.Second) http_helper.HttpGetWithRetry(t, serviceUrl(t, opts), // URL to test 200, // Expected status code "Hello, World!", // Expected body 10, // Max retries 3*time.Second // Time between retries ) } 1. Wait until the service is deployed 92. func validate(t *testing.T, opts *k8s.KubectlOptions) { k8s.WaitUntilServiceAvailable(t, opts, "hello-world- app-service", 10, 1*time.Second) http_helper.HttpGetWithRetry(t, serviceUrl(t, opts), // URL to test 200, // Expected status code "Hello, World!", // Expected body 10, // Max retries 3*time.Second // Time between retries ) } 2. Make HTTP requests 93. func validate(t *testing.T, opts *k8s.KubectlOptions) { k8s.WaitUntilServiceAvailable(t, opts, "hello-world- app-service", 10, 1*time.Second) http_helper.HttpGetWithRetry(t, serviceUrl(t, opts), // URL to test 200, // Expected status code "Hello, World!", // Expected body 10, // Max retries 3*time.Second // Time between retries ) } 3. Use serviceUrl method to get URL 94. func serviceUrl(t *testing.T, opts *k8s.KubectlOptions) string { service := k8s.GetService(t, options, "hello-world-app-service") endpoint := k8s.GetServiceEndpoint(t, options, service, 8080) return fmt.Sprintf("http://%s", endpoint) } The serviceUrl method 95. $ kubectl config set-credentials â To run the test, first authenticate to a Kubernetes cluster. 96. Note: Kubernetes is now part of Docker Desktop. Test 100% locally! 97. $ go test -v -timeout 15m -run TestDockerKubernetes â --- PASS: TestDockerKubernetes (5.69s) Run go test. You can validate your config after every commit in seconds! 98. Unit tests 1. Unit testing basics 2. Example: Terraform unit tests 3. Example: Docker/Kubernetes unit tests 4. Cleaning up after tests 99. Note: tests create and destroy many resources! 100. Pro tip #1: run tests in completely separate âsandboxâ accounts 101. Tool Clouds Features cloud-nuke AWS (GCP planned) Delete all resources older than a certain date; in a certain region; of a certain type. Janitor Monkey AWS Configurable rules of what to delete. Notify owners of pending deletions. aws-nuke AWS Specify specific AWS accounts and resource types to target. Azure Powershell Azure Includes native commands to delete Resource Groups Pro tip #2: run these tools in cron jobs to clean up left-over resources 102. 1. Static analysis 2. Unit tests 3. Integration tests 4. End-to-end tests 5. Conclusion Outline 103. Integration tests: test multiple âunitsâ work together. 104. Integration tests 1. Example: Terraform integration tests 2. Test parallelism 3. Test stages 4. Test retries 105. Integration tests 1. Example: Terraform integration tests 2. Test parallelism 3. Test stages 4. Test retries 106. infrastructure-as-code-testing-talk └ examples └ hello-world-app └ docker-kubernetes └ proxy-app └ web-service └ modules └ test └ README.md Letâs say you have two Terraform modules you want to test together: 107. infrastructure-as-code-testing-talk └ examples └ hello-world-app └ docker-kubernetes └ proxy-app └ web-service └ modules └ test └ README.md proxy-app: an app that acts as an HTTP proxy for other web services. 108. infrastructure-as-code-testing-talk └ examples └ hello-world-app └ docker-kubernetes └ proxy-app └ web-service └ modules └ test └ README.md web-service: a web service that you want proxied. 109. variable "url_to_proxy" { description = "The URL to proxy." type = string } proxy-app takes in the URL to proxy via an input variable 110. output "url" { value = module.web_service.url } web-service exposes its URL via an output variable 111. infrastructure-as-code-testing-talk └ examples └ modules └ test └ hello_world_app_test.go └ docker_kubernetes_test.go └ proxy_app_test.go └ README.md Create proxy_app_test.go 112. func TestProxyApp(t *testing.T) { webServiceOpts := configWebService(t) defer terraform.Destroy(t, webServiceOpts) terraform.InitAndApply(t, webServiceOpts) proxyAppOpts := configProxyApp(t, webServiceOpts) defer terraform.Destroy(t, proxyAppOpts) terraform.InitAndApply(t, proxyAppOpts) validate(t, proxyAppOpts) } The basic test structure 113. func TestProxyApp(t *testing.T) { webServiceOpts := configWebService(t) defer terraform.Destroy(t, webServiceOpts) terraform.InitAndApply(t, webServiceOpts) proxyAppOpts := configProxyApp(t, webServiceOpts) defer terraform.Destroy(t, proxyAppOpts) terraform.InitAndApply(t, proxyAppOpts) validate(t, proxyAppOpts) } 1. Configure options for the web service 114. func TestProxyApp(t *testing.T) { webServiceOpts := configWebService(t) defer terraform.Destroy(t, webServiceOpts) terraform.InitAndApply(t, webServiceOpts) proxyAppOpts := configProxyApp(t, webServiceOpts) defer terraform.Destroy(t, proxyAppOpts) terraform.InitAndApply(t, proxyAppOpts) validate(t, proxyAppOpts) } 2. Deploy the web service 115. func TestProxyApp(t *testing.T) { webServiceOpts := configWebService(t) defer terraform.Destroy(t, webServiceOpts) terraform.InitAndApply(t, webServiceOpts) proxyAppOpts := configProxyApp(t, webServiceOpts) defer terraform.Destroy(t, proxyAppOpts) terraform.InitAndApply(t, proxyAppOpts) validate(t, proxyAppOpts) } 3. Configure options for the proxy app (passing it the web service options) 116. func TestProxyApp(t *testing.T) { webServiceOpts := configWebService(t) defer terraform.Destroy(t, webServiceOpts) terraform.InitAndApply(t, webServiceOpts) proxyAppOpts := configProxyApp(t, webServiceOpts) defer terraform.Destroy(t, proxyAppOpts) terraform.InitAndApply(t, proxyAppOpts) validate(t, proxyAppOpts) } 4. Deploy the proxy app 117. func TestProxyApp(t *testing.T) { webServiceOpts := configWebService(t) defer terraform.Destroy(t, webServiceOpts) terraform.InitAndApply(t, webServiceOpts) proxyAppOpts := configProxyApp(t, webServiceOpts) defer terraform.Destroy(t, proxyAppOpts) terraform.InitAndApply(t, proxyAppOpts) validate(t, proxyAppOpts) } 5. Validate the proxy app works 118. func TestProxyApp(t *testing.T) { webServiceOpts := configWebService(t) defer terraform.Destroy(t, webServiceOpts) terraform.InitAndApply(t, webServiceOpts) proxyAppOpts := configProxyApp(t, webServiceOpts) defer terraform.Destroy(t, proxyAppOpts) terraform.InitAndApply(t, proxyAppOpts) validate(t, proxyAppOpts) } 6. At the end of the test, undeploy the proxy app and the web service 119. func configWebService(t *testing.T) *terraform.Options { return &terraform.Options{ TerraformDir: "../examples/web-service", } } The configWebService method 120. func configProxyApp(t *testing.T, webServiceOpts *terraform.Options) *terraform.Options { url := terraform.Output(t, webServiceOpts, "url") return &terraform.Options{ TerraformDir: "../examples/proxy-app", Vars: map[string]interface{}{ "url_to_proxy": url, }, } } The configProxyApp method 121. func configProxyApp(t *testing.T, webServiceOpts *terraform.Options) *terraform.Options { url := terraform.Output(t, webServiceOpts, "url") return &terraform.Options{ TerraformDir: "../examples/proxy-app", Vars: map[string]interface{}{ "url_to_proxy": url, }, } } 1. Read the url output from the web- service module 122. func configProxyApp(t *testing.T, webServiceOpts *terraform.Options) *terraform.Options { url := terraform.Output(t, webServiceOpts, "url") return &terraform.Options{ TerraformDir: "../examples/proxy-app", Vars: map[string]interface{}{ "url_to_proxy": url, }, } } 2. Pass it in as the url_to_proxy input to the proxy-app module 123. func validate(t *testing.T, opts *terraform.Options) { url := terraform.Output(t, opts, "url") http_helper.HttpGetWithRetry(t, url, // URL to test 200, // Expected status code `{"text":"Hello, World!"}`, // Expected body 10, // Max retries 3 * time.Second // Time between retries ) } The validate method 124. $ go test -v -timeout 15m -run TestProxyApp â --- PASS: TestProxyApp (182.44s) Run go test. Youâre now testing multiple modules together! 125. $ go test -v -timeout 15m -run TestProxyApp â --- PASS: TestProxyApp (182.44s) But integration tests can take (many) minutes to runâ 126. Integration tests 1. Example: Terraform integration tests 2. Test parallelism 3. Test stages 4. Test retries 127. Infrastructure tests can take a long time to run 128. One way to save time: run tests in parallel 129. func TestProxyApp(t *testing.T) { t.Parallel() // The rest of the test code } func TestHelloWorldAppUnit(t *testing.T) { t.Parallel() // The rest of the test code } Enable test parallelism in Go by adding t.Parallel() as the 1st line of each test. 130. $ go test -v -timeout 15m === RUN TestHelloWorldApp === RUN TestDockerKubernetes === RUN TestProxyApp Now, if you run go test, all the tests with t.Parallel() will run in parallel 131. But thereâs a gotcha: resource conflicts 132. resource "aws_iam_role" "role_example" { name = "example-iam-role" } resource "aws_security_group" "sg_example" { name = "security-group-example" } Example: module with hard-coded IAM Role and Security Group names 133. resource "aws_iam_role" "role_example" { name = "example-iam-role" } resource "aws_security_group" "sg_example" { name = "security-group-example" } If two tests tried to deploy this module in parallel, the names would conflict! 134. Key takeaway: you must namespace all your resources 135. resource "aws_iam_role" "role_example" { name = var.name } resource "aws_security_group" "sg_example" { name = var.name } Example: use variables in all resource namesâ 136. uniqueId := random.UniqueId() return &terraform.Options{ TerraformDir: "../examples/proxy-app", Vars: map[string]interface{}{ "name": fmt.Sprintf("text-proxy-app-%s", uniqueId) }, } At test time, set the variables to a randomized value to avoid conflicts 137. Integration tests 1. Example: Terraform integration tests 2. Test parallelism 3. Test stages 4. Test retries 138. Consider the structure of the proxy-app integration test: 139. 1. Deploy web-service 2. Deploy proxy-app 3. Validate proxy-app 4. Undeploy proxy-app 5. Undeploy web-service 140. 1. Deploy web-service 2. Deploy proxy-app 3. Validate proxy-app 4. Undeploy proxy-app 5. Undeploy web-service When iterating locally, you sometimes want to re-run just one of these steps. 141. 1. Deploy web-service 2. Deploy proxy-app 3. Validate proxy-app 4. Undeploy proxy-app 5. Undeploy web-service But as the code is written now, you have to run all steps on each test run. 142. 1. Deploy web-service 2. Deploy proxy-app 3. Validate proxy-app 4. Undeploy proxy-app 5. Undeploy web-service And that can add up to a lot of overhead. (~3 min) (~2 min) (~30 seconds) (~1 min) (~2 min) 143. Key takeaway: break your tests into independent test stages 144. webServiceOpts := configWebService(t) defer terraform.Destroy(t, webServiceOpts) terraform.InitAndApply(t, webServiceOpts) proxyAppOpts := configProxyApp(t, webServiceOpts) defer terraform.Destroy(t, proxyAppOpts) terraform.InitAndApply(t, proxyAppOpts) validate(t, proxyAppOpts) The original test structure 145. stage := test_structure.RunTestStage defer stage(t, "cleanup_web_service", cleanupWebService) stage(t, "deploy_web_service", deployWebService) defer stage(t, "cleanup_proxy_app", cleanupProxyApp) stage(t, "deploy_proxy_app", deployProxyApp) stage(t, "validate", validate) The test structure with test stages 146. stage := test_structure.RunTestStage defer stage(t, "cleanup_web_service", cleanupWebService) stage(t, "deploy_web_service", deployWebService) defer stage(t, "cleanup_proxy_app", cleanupProxyApp) stage(t, "deploy_proxy_app", deployProxyApp) stage(t, "validate", validate) 1. RunTestStage is a helper function from Terratest. 147. stage := test_structure.RunTestStage defer stage(t, "cleanup_web_service", cleanupWebService) stage(t, "deploy_web_service", deployWebService) defer stage(t, "cleanup_proxy_app", cleanupProxyApp) stage(t, "deploy_proxy_app", deployProxyApp) stage(t, "validate", validate) 2. Wrap each stage of your test with a call to RunTestStage 148. stage := test_structure.RunTestStage defer stage(t, "cleanup_web_service", cleanupWebService) stage(t, "deploy_web_service", deployWebService) defer stage(t, "cleanup_proxy_app", cleanupProxyApp) stage(t, "deploy_proxy_app", deployProxyApp) stage(t, "validate", validate) 3. Define each stage in a function (youâll see this code shortly). 149. stage := test_structure.RunTestStage defer stage(t, "cleanup_web_service", cleanupWebService) stage(t, "deploy_web_service", deployWebService) defer stage(t, "cleanup_proxy_app", cleanupProxyApp) stage(t, "deploy_proxy_app", deployProxyApp) stage(t, "validate", validate) 4. Give each stage a unique name 150. stage := test_structure.RunTestStage defer stage(t, "cleanup_web_service", cleanupWebService) stage(t, "deploy_web_service", deployWebService) defer stage(t, "cleanup_proxy_app", cleanupProxyApp) stage(t, "deploy_proxy_app", deployProxyApp) stage(t, "validate", validate) Any stage foo can be skipped by setting the env var SKIP_foo=true 151. $ SKIP_cleanup_web_service=true $ SKIP_cleanup_proxy_app=true Example: on the very first test run, skip the cleanup stages. 152. $ go test -v -timeout 15m -run TestProxyApp Running stage 'deploy_web_service'â Running stage 'deploy_proxy_app'â Running stage 'validate'â Skipping stage 'cleanup_proxy_app'â Skipping stage 'cleanup_web_service'â --- PASS: TestProxyApp (105.73s) That way, after the test finishes, the infrastructure will still be running. 153. $ SKIP_deploy_web_service=true $ SKIP_deploy_proxy_app=true Now, on the next several test runs, you can skip the deploy stages too. 154. $ go test -v -timeout 15m -run TestProxyApp Skipping stage 'deploy_web_serviceââ Skipping stage 'deploy_proxy_app'â Running stage 'validate'â Skipping stage 'cleanup_proxy_app'â Skipping stage 'cleanup_web_service'â --- PASS: TestProxyApp (14.22s) This allows you to iterate on solely the validate stageâ 155. $ go test -v -timeout 15m -run TestProxyApp Skipping stage 'deploy_web_serviceââ Skipping stage 'deploy_proxy_app'â Running stage 'validate'â Skipping stage 'cleanup_proxy_app'â Skipping stage 'cleanup_web_service'â --- PASS: TestProxyApp (14.22s) Which dramatically speeds up your iteration / feedback cycle! 156. $ SKIP_validate=true $ unset SKIP_cleanup_web_service $ unset SKIP_cleanup_proxy_app When youâre done iterating, skip validate and re-enable cleanup 157. $ go test -v -timeout 15m -run TestProxyApp Skipping stage 'deploy_web_serviceââ Skipping stage 'deploy_proxy_appââ Skipping stage 'validateââ Running stage 'cleanup_proxy_appââ Running stage 'cleanup_web_service'â --- PASS: TestProxyApp (59.61s) This cleans up everything that was left running. 158. func deployWebService(t *testing.T) { opts := configWebServiceOpts(t) test_structure.SaveTerraformOptions(t, "/tmp", opts) terraform.InitAndApply(t, opts) } func cleanupWebService(t *testing.T) { opts := test_structure.LoadTerraformOptions(t, "/tmp") terraform.Destroy(t, opts) } Note: each time you run test stages via go test, itâs a separate OS process. 159. func deployWebService(t *testing.T) { opts := configWebServiceOpts(t) test_structure.SaveTerraformOptions(t, "/tmp", opts) terraform.InitAndApply(t, opts) } func cleanupWebService(t *testing.T) { opts := test_structure.LoadTerraformOptions(t, "/tmp") terraform.Destroy(t, opts) } So to pass data between stages, one stage needs to write the data to diskâ 160. func deployWebService(t *testing.T) { opts := configWebServiceOpts(t) test_structure.SaveTerraformOptions(t, "/tmp", opts) terraform.InitAndApply(t, opts) } func cleanupWebService(t *testing.T) { opts := test_structure.LoadTerraformOptions(t, "/tmp") terraform.Destroy(t, opts) } And the other stages need to read that data from disk. 161. Integration tests 1. Example: Terraform integration tests 2. Test parallelism 3. Test stages 4. Test retries 162. Real infrastructure can fail for intermittent reasons (e.g., bad EC2 instance, Apt downtime, Terraform bug) 163. To avoid âflakyâ tests, add retries for known errors. 164. &terraform.Options{ TerraformDir: "../examples/proxy-app", RetryableTerraformErrors: map[string]string{ "net/http: TLS handshake timeout": "Terraform bug", }, MaxRetries: 3, TimeBetweenRetries: 3*time.Second, } Example: retry up to 3 times on a known TLS error in Terraform. 165. 1. Static analysis 2. Unit tests 3. Integration tests 4. End-to-end tests 5. Conclusion Outline 166. End-to-end tests: test your entire infrastructure works together. 167. How do you test this entire thing? 168. You could use the same strategyâ 1. Deploy all the infrastructure 2. Validate it works (e.g., via HTTP requests, API calls, SSH commands, etc.) 3. Undeploy all the infrastructure 169. But itâs rare to write end-to- end tests this way. Hereâs why: 170. e2e Tests Test pyramid Integration Tests Unit Tests Static analysis 171. e2e Tests Integration Tests Unit Tests Static analysis Cost, brittleness, run time 172. e2e Tests Integration Tests Unit Tests Static analysis 60 â 240+ minutes 5 â 60 minutes 1 â 20 minutes 1 â 60 seconds 173. e2e Tests Integration Tests Unit Tests Static analysis E2E tests are too slow to be useful 60 â 240+ minutes 5 â 60 minutes 1 â 20 minutes 1 â 60 seconds 174. Another problem with E2E tests: brittleness. 175. Letâs do some math: 176. Assume a single resource (e.g., EC2 instance) has a 1/1000 (0.1%) chance of failure. 177. Test type # of resources Chance of failure Unit tests 10 1% Integration tests 50 5% End-to-end tests 500+ 40%+ The more resources your tests deploy, the flakier they will be. 178. Test type # of resources Chance of failure Unit tests 10 1% Integration tests 50 5% End-to-end tests 500+ 40%+ You can work around the failure rate for unit & integration tests with retries 179. Test type # of resources Chance of failure Unit tests 10 1% Integration tests 50 5% End-to-end tests 500+ 40%+ You can work around the failure rate for unit & integration tests with retries 180. Key takeaway: E2E tests from scratch are too slow and too brittle to be useful 181. Instead, you can do incremental E2E testing! 182. module module module module module module module module module module module module module module module 1. Deploy a persistent test environment and leave it running. 183. module module module module module module module module module module module module module module module 2. Each time you update a module, deploy & validate just that module 184. module module module module module module module module module module module module module module module 3. Bonus: test your deployment process is zero-downtime too! 185. 1. Static analysis 2. Unit tests 3. Integration tests 4. End-to-end tests 5. Conclusion Outline 186. Testing techniques compared: 187. Technique Strengths Weaknesses Static analysis 1. Fast 2. Stable 3. No need to deploy real resources 4. Easy to use 1. Very limited in errors you can catch 2. You donât get much confidence in your code solely from static analysis Unit tests 1. Fast enough (1 â 10 min) 2. Mostly stable (with retry logic) 3. High level of confidence in individual units 1. Need to deploy real resources 2. Requires writing non-trivial code Integration tests 1. Mostly stable (with retry logic) 2. High level of confidence in multiple units working together 1. Need to deploy real resources 2. Requires writing non-trivial code 3. Slow (10 â 30 min) End-to-end tests 1. Build confidence in your entire architecture 1. Need to deploy real resources 2. Requires writing non-trivial code 3. Very slow (60 min â 240+ min)* 4. Can be brittle (even with retry logic)* 188. So which should you use? 189. All of them! They all catch different types of bugs. 190. e2e Tests Keep in mind the test pyramid Integration Tests Unit Tests Static analysis 191. e2e Tests Lots of unit tests + static analysis Integration Tests Unit Tests Static analysis 192. e2e Tests Fewer integration tests Integration Tests Unit Tests Static analysis 193. e2e Tests A handful of high-value e2e tests Integration Tests Unit Tests Static analysis 194. Infrastructure code without tests is scary 195. Fight the fear & build confidence in your code with automated tests 196. Questions? info@gruntwork.io https://www.friendbookmark.com/videos/965/kvm-and-docker-lxc-benchmarking-with-openstackPassive benchmarking with docker LXC and KVM using OpenStack hosted in SoftLayer. These results provide initial incite as to why LXC as a technology choice offers benefits over traditional VMs and seek to provide answers as to the typical initial LXC question -- "why would I consider Linux Containers over VMs" from a performance perspective. Results here provide insight as to: - Cloudy ops times (start, stop, reboot) using OpenStack. - Guest micro benchmark performance (I/O, network, memory, CPU). - Guest micro benchmark performance of MySQL; OLTP read, read / write complex and indexed insertion. - Compute node resource consumption; VM / Container density factors. - Lessons learned during benchmarking. The tests here were performed using OpenStack Rally to drive the OpenStack cloudy tests and various other linux tools to test the guest performance on a "micro level". The nova docker virt driver was used in the Cloud scenario to realize VMs as docker LXC containers and compared to the nova virt driver for libvirt KVM. Please read the disclaimers in the presentation as this is only intended to be the "chip of the ice burg". Topics covered in this presentation slides: 1. Passive Benchmarking with docker LXC, KVM & OpenStack Hosted @ SoftLayer Boden Russell (brussell@us.ibm.com) IBM Global Technology Services Advanced Cloud Solutions & Innovation V2.0 2. FAQ - How is this version (v2.0) different from the initial benchmarks? â See the revision history within this document. - Are there any artifacts associated with the test? â Yes; see my github repo: https://github.com/bodenr/cloudy-docker-kvm-bench - Do these results imply an LXC based technology replaces the need for traditional hypervisors? â In my opinion, traditional VMs will become the âedge caseâ moving forward for use cases which are currently based on Linux flavored VMs. However I believe there will still be cases for traditional VMs, some of which are detailed in the LXC Realization presentation. - Are these results scientific? â No. Disclaimers have been attached to any documentation related to these tests to indicate such. These tests are meant to be a set of âlitmusâ tests to gain an initial understanding of how LXC compares to traditional hypervisors specifically in the Cloud space. - Do you welcome comments / feedback on the test? â Yes; the goal of these tests is to educate the community on LXC based technologies vs traditional hypervisors. As such they are fully disclosed in complete and hence open to feedback of any kind. 5/11/2014 2Document v2.0 3. FAQ Continued - Should I act on these results? â I believe the results provide enough information to gain some interest. I expect any organization, group or individual considering actions as a result will perform their own validation to assert the technology choice is beneficial for their consumption prior to adoption. - Is further / deeper testing and investigation warranted? â Absolutely. These tests should be conducted in a more active manner to understand the root causes for any differences. Additional tests and variations are also needed including; various KVM disk cache modes, skinny VM images (i.e. JeOS), impacts of database settings, docker storage drivers, etc. - Is this a direct measurement of the hypervisor (KVM) or LXC engine (docker)? â No, many factors play into results. For example the compute node has the nova virt driver running which is obviously different in implementation between nova libvirt-kvm and nova docker. Thus itâs implementation *may* have an impact on the compute node metrics and performance. 5/11/2014 Document v2.0 3 4. Revision History Revision Overview of changes V1.0 - Initial document release V2.0 - All tests were re-run using a single docker image throughout the tests (see my Dockerfile). - As the result of an astute reader, the 15 VM serial âpackingâ test reflects VM boot overhead rather than steady- state; this version clarifies such claims. - A new Cloudy test was added to better understand steady-state CPU. - Rather than presenting direct claims of density, raw data and graphs are presented to let the reader draw their own conclusions. - Additional âin the guestâ tests were performed including blogbench. 5/11/2014 Document v2.0 4 5. Why Linux Containers (LXC) - Fast â Runtime performance near bare metal speeds â Management operations (run, stop , start, etc.) in seconds / milliseconds - Agile â VM-like agility â itâs still âvirtualizationâ â Seamlessly âmigrateâ between virtual and bare metal environments - Flexible â Containerize a âsystemâ â Containerize âapplication(s)â - Lightweight â Just enough Operating System (JeOS) â Minimal per container penalty - Inexpensive â Open source â free â lower TCO â Supported with out-of-the-box modern Linux kernel - Ecosystem â Growing in popularity â Vibrant community & numerous 3rd party apps 5/11/2014 5Document v2.0 6. Hypervisors vs. Linux Containers Hardware Operating System Hypervisor Virtual Machine Operating System Bins / libs App App Virtual Machine Operating System Bins / libs App App Hardware Hypervisor Virtual Machine Operating System Bins / libs App App Virtual Machine Operating System Bins / libs App App Hardware Operating System Container Bins / libs App App Container Bins / libs App App Type 1 Hypervisor Type 2 Hypervisor Linux Containers 5/11/2014 6 Containers share the OS kernel of the host and thus are lightweight. However, each container must have the same OS kernel. Containers are isolated, but share OS and, where appropriate, libs / bins. Document v2.0 7. Hypervisor VM vs. LXC vs. Docker LXC 5/11/2014 7Document v2.0 8. Docker in OpenStack - Havana â Nova virt driver which integrates with docker REST API on backend â Glance translator to integrate docker images with Glance - Icehouse â Heat plugin for docker - Both options are still under development 5/11/2014 8 nova-docker virt driver docker heat plugin DockerInc::Docke r::Container (plugin) Document v2.0 9. About This Benchmark - Use case perspective â As an OpenStack Cloud user I want a Ubuntu based VM with MySQLâ Why would I choose docker LXC vs a traditional hypervisor? - OpenStack âCloudyâ perspective â LXC vs. traditional VM from a Cloudy (OpenStack) perspective â VM operational times (boot, start, stop, snapshot) â Compute node resource usage (per VM penalty); density factor - Guest runtime perspective â CPU, memory, file I/O, MySQL OLTP, etc. - Why KVM? â Exceptional performance DISCLAIMERS The tests herein are semi-active litmus tests â no in depth tuning, analysis, etc. More active testing is warranted. These results do not necessary reflect your workload or exact performance nor are they guaranteed to be statistically sound. 5/11/2014 9Document v2.0 10. Benchmark Environment Topology @ SoftLayer glance api / reg nova api / cond / etc keystone â rally nova api / cond / etc cinder api / sch / vol docker lxc dstat controller compute node glance api / reg nova api / cond / etc keystone â rally nova api / cond / etc cinder api / sch / vol KVM dstat controller compute node 5/11/2014 10 + Awesome! + Awesome! Document v2.0 11. Benchmark Specs 5/11/2014 11 Spec Controller Node (4CPU x 8G RAM) Compute Node (16CPU x 96G RAM) Environment Bare Metal @ SoftLayer Bare Metal @ SoftLayer Mother Board SuperMicro X8SIE-F Intel Xeon QuadCore SingleProc SATA [1Proc] SuperMicro X8DTU-F_R2 Intel Xeon HexCore DualProc [2Proc] CPU Intel Xeon-Lynnfield 3470-Quadcore [2.93GHz] (Intel Xeon-Westmere 5620-Quadcore [2.4GHz]) x 2 Memory (Kingston 4GB DDR3 2Rx8 4GB DDR3 2Rx8 [4GB]) x2 (Kingston 16GB DDR3 2Rx4 16GB DDR3 2Rx4 [16GB]) x 6 HDD (LOCAL) Digital WD Caviar RE3 WD5002ABYS [500GB]; SATAII Western Digital WD Caviar RE4 WD5003ABYX [500GB]; SATAII NIC eth0/eth1 @ 100 Mbps eth0/eth1 @100 Mbps Operating System Ubuntu 12.04 LTS 64bit Ubuntu 12.04 LTS 64bit Kernel 3.5.0-48-generic 3.8.0-38-generic IO Scheduler deadline deadline Hypervisor tested NA - KVM 1.0 + virtio + KSM (memory deduplication) - docker 0.10.0 + go1.2.1 + commit dc9c28f + AUFS OpenStack Trunk master via devstack Trunk master via devstack. Libvirt KVM nova driver / nova-docker virt driver OpenStack Benchmark Client OpenStack project rally NA Metrics Collection NA dstat Guest Benchmark Driver NA - Sysbench 0.4.12 - mbw 1.1.1.-2 - iibench (py) - netperf 2.5.0-1 - Blogbench 1.1 - cpu_bench.py VM Image NA - Scenario 1 (KVM): official ubuntu 12.04 image + mysql snapshotted and exported to qcow2 â 1080 MB - Scenario 2 (docker): guillermo/mysql -- 381.5 MB Hosted @Document v2.0 12. Test Descriptions: Cloudy Benchmarks 5/11/2014 12 Benchmark Benchmark Driver Description OpenStack Cloudy Benchmarks Serial VM boot (15 VMs) OpenStack Rally - Boot VM from image - Wait for ACTIVE state - Repeat the above a total of 15 times - Delete VMs Compute node steady-state VM packing cpu_bench.py - Boot 15 VMs in async fashion - Sleep for 5 minutes (wait for steady-state) - Delete all 15 VMs in async fashion VM reboot (5 VMs rebooted 5 times each) OpenStack Rally - Boot VM from image - Wait for ACTIVE state - Soft reboot VM 5 times - Delete VM - Repeat the above a total of 5 times VM snapshot (1 VM, 1 snapshot) OpenStack Rally - Boot VM from image - Wait for ACTIVE state - Snapshot VM to glance image - Delete VM Document v2.0 13. Test Descriptions: Guest Benchmarks 5/11/2014 13 Benchmark Benchmark Driver Description Guest Runtime Benchmarks CPU performance Sysbench from within the guest - Clear memory cache - Run sysbench cpu test - Repeat a total of 5 times - Average results over the 5 times OLTP (MySQL) performance Sysbench from within the guest - Clear memory cache - Run sysbench OLTP test - Repeat a total of 5 times - Average results over the 5 times MySQL Indexed insertion benchmark - Clear memory cache - Run iibench for a total of 1M inserts printing stats at 100K intervals - Collect data over 5 runs & average File I/O performance Sysbench from within the guest - Synchronous IO - Clear memory cache - Run sysbench OLTP test - Repeat a total of 5 times - Average results over the 5 times Memory performance Mbw from within the guest - Clear memory cache - Run mbw with array size of 1000 MiB and each test 10 times - Collect average over 10 runs per test Network performance Netperf - Run netperf server on controller - From guest run netperf client in IPv4 mode - Repeat text 5x - Average results Application type performance Blogbench - Clear memory cache - Run blogbench for 5 minutes - Repeat 5 times - Average read / write scores Document v2.0 14. STEADY STATE VM PACKING OpenStack Cloudy Benchmark 5/11/2014 14Document v2.0 15. Cloudy Performance: Steady State Packing - Benchmark scenario overview â Pre-cache VM image on compute node prior to test â Boot 15 VM asynchronously in succession â Wait for 5 minutes (to achieve steady-state on the compute node) â Delete all 15 VMs asynchronously in succession - Benchmark driver â cpu_bench.py - High level goals â Understand compute node characteristics under steady-state conditions with 15 packed / active VMs 5/11/2014 15 0 2 4 6 8 10 12 14 16 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 ActiveVMs Time Benchmark Visualization VMs Document v2.0 16. Cloudy Performance: Steady State Packing 5/11/2014 16 0 10 20 30 40 50 60 70 80 1 9 17 25 33 41 49 57 65 73 81 89 97 105 113 121 129 137 145 153 161 169 177 185 193 201 209 217 225 233 241 249 257 265 273 281 289 297 305 313 321 CPUUsageInPercent Time Docker: Compute Node CPU (full test duration) usr sys Averages â 0.54 â 0.17 0 10 20 30 40 50 60 70 80 1 9 17 25 33 41 49 57 65 73 81 89 97 105 113 121 129 137 145 153 161 169 177 185 193 201 209 217 225 233 241 249 257 265 273 281 289 297 305 313 321 329 337 345 CPUUsageInPercent Time KVM: Compute Node CPU (full test duration) usr sys Averages â 7.64 â 1.4 Document v2.0 17. Cloudy Performance: Steady State Packing 5/11/2014 17 0 2 4 6 8 10 12 14 1 6 11 16 21 26 31 36 41 46 51 56 61 66 71 76 81 86 91 96 101 106 111 116 121 126 131 136 141 146 151 156 161 166 171 176 181 186 191 196 201 206 211 CPUUsageInPercent Time (31s â 243s) Docker: Compute Node Steady-State CPU (segment: 31s â 243s) usr sys 0 2 4 6 8 10 12 14 1 6 11 16 21 26 31 36 41 46 51 56 61 66 71 76 81 86 91 96 101 106 111 116 121 126 131 136 141 146 151 156 161 166 171 176 181 186 191 196 201 206 211 CPUUsageInPercent Time (95s - 307s) KVM: Compute Node Steady-State CPU (segment: 95s â 307s) usr sys Averages â 0.2 â 0.03 Averages â 1.91 â 0.36 31 seconds 243 seconds 95 seconds 307 seconds Document v2.0 18. Cloudy Performance: Steady State Packing 5/11/2014 18 0 2 4 6 8 10 12 14 1 7 13 19 25 31 37 43 49 55 61 67 73 79 85 91 97 103 109 115 121 127 133 139 145 151 157 163 169 175 181 187 193 199 205 211 CPUUsageInPercent Time: KVM(95s - 307s) Docker(31s â 243s) Docker / KVM: Compute Node Steady-State CPU (Segment Overlay) docker-usr docker-sys kvm-usr kvm-sys docker: 31s KVM: 95s docker: 243s KVM: 307s Docker Averages â 0.2 â 0.03 KVM Averages â 1.91 â 0.36 Document v2.0 19. Cloudy Performance: Steady State Packing 5/11/2014 19 0.00E+00 1.00E+09 2.00E+09 3.00E+09 4.00E+09 5.00E+09 6.00E+09 7.00E+09 1 9 17 25 33 41 49 57 65 73 81 89 97 105 113 121 129 137 145 153 161 169 177 185 193 201 209 217 225 233 241 249 257 265 273 281 289 297 305 313 321 MemoryUsed Time Docker: Compute Node Used Memory (full test duration) Memory Delta 734 MB Per VM 49 MB 0.00E+00 1.00E+09 2.00E+09 3.00E+09 4.00E+09 5.00E+09 6.00E+09 7.00E+09 1 10 19 28 37 46 55 64 73 82 91 100 109 118 127 136 145 154 163 172 181 190 199 208 217 226 235 244 253 262 271 280 289 298 307 316 325 334 MemoryUsed Time KVM: Compute Node Used Memory (full test duration) Memory Delta 4387 MB Per VM 292 MB Document v2.0 20. Cloudy Performance: Steady State Packing 5/11/2014 20 0.00E+00 1.00E+09 2.00E+09 3.00E+09 4.00E+09 5.00E+09 6.00E+09 7.00E+09 1 10 19 28 37 46 55 64 73 82 91 100 109 118 127 136 145 154 163 172 181 190 199 208 217 226 235 244 253 262 271 280 289 298 307 316 325 334 MemoryUsed Axis Title Docker / KVM: Compute Node Used Memory (Overlay) kvm docker Document v2.0 21. Cloudy Performance: Steady State Packing 5/11/2014 21 0 10 20 30 40 50 60 70 80 90 100 1 9 17 25 33 41 49 57 65 73 81 89 97 105 113 121 129 137 145 153 161 169 177 185 193 201 209 217 225 233 241 249 257 265 273 281 289 297 305 313 321 1MinuteLoadAverage Time Docker: Compute Node 1m Load Average (full test duration) 1m Average 0.15 % 0 10 20 30 40 50 60 70 80 90 100 1 9 17 25 33 41 49 57 65 73 81 89 97 105 113 121 129 137 145 153 161 169 177 185 193 201 209 217 225 233 241 249 257 265 273 281 289 297 305 313 321 329 337 1MinuteLoadAverage Time KVM: Compute Node 1m Load Average (full test duration) 1m Average 35.9 % Document v2.0 22. SERIALLY BOOT 15 VMS OpenStack Cloudy Benchmark 5/11/2014 22Document v2.0 23. Cloudy Performance: Serial VM Boot - Benchmark scenario overview â Pre-cache VM image on compute node prior to test â Boot VM â Wait for VM to become ACTIVE â Repeat the above steps for a total of 15 VMs â Delete all VMs - Benchmark driver â OpenStack Rally - High level goals â Understand compute node characteristics under sustained VM boots 5/11/2014 23 0 2 4 6 8 10 12 14 16 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 ActiveVMs Time Benchmark Visualization VMs Document v2.0 24. Cloudy Performance: Serial VM Boot 5/11/2014 24 3.529113102 5.781662448 0 1 2 3 4 5 6 7 docker KVM TimeInSeconds Average Server Boot Time docker KVM Document v2.0 25. Cloudy Performance: Serial VM Boot 5/11/2014 25 0 5 10 15 20 25 30 35 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 67 69 71 73 75 77 79 CPUUsageInPercent Time Docker: Compute Node CPU usr sys Averages â 1.39 â 0.57 0 5 10 15 20 25 30 35 1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 70 73 76 79 82 85 88 91 94 97 100 103 106 109 112 115 118 121 124 127 CPUUsageInPercent Time KVM: Compute Node CPU Usage usr sys Averages â 13.45 â 2.23 Document v2.0 26. Cloudy Performance: Serial VM Boot 5/11/2014 26 0 5 10 15 20 25 30 35 1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97 101105109113117121125 CPUUsageInPercent Time Docker / KVM: Compute Node CPU (Unnormalized Overlay) kvm-usr kvm-sys docker-usr docker-sys Document v2.0 27. Cloudy Performance: Serial VM Boot 5/11/2014 27 y = 0.0095x + 1.008 y = 0.3582x + 1.0633 0 5 10 15 20 25 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 UsrCPUInPercent Time (8s - 58s) Docker / KVM: Serial VM Boot Usr CPU (segment: 8s - 58s) docker(8-58) kvm(8-58) Linear (docker(8-58)) Linear (kvm(8-58)) 8 seconds 58 seconds Document v2.0 28. Cloudy Performance: Serial VM Boot 5/11/2014 28 0.00E+00 5.00E+08 1.00E+09 1.50E+09 2.00E+09 2.50E+09 3.00E+09 3.50E+09 4.00E+09 4.50E+09 5.00E+09 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 67 69 71 73 75 77 79 MemoryUsed Time Docker: Compute Node Memory Used Memory Delta 677 MB Per VM 45 MB 0.00E+00 1.00E+09 2.00E+09 3.00E+09 4.00E+09 5.00E+09 1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97 101105109113117121125 MemoryUsed Time KVM: Compute Node Memory Used Memory Delta 2737 MB Per VM 182 MB Document v2.0 29. Cloudy Performance: Serial VM Boot 5/11/2014 29 0.00E+00 5.00E+08 1.00E+09 1.50E+09 2.00E+09 2.50E+09 3.00E+09 3.50E+09 4.00E+09 4.50E+09 5.00E+09 1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97 101105109113117121125 MemoryUsed Time Docker / KVM: Compute Node Memory Used (Unnormalized Overlay) kvm docker Document v2.0 30. Cloudy Performance: Serial VM Boot 5/11/2014 30 y = 1E+07x + 1E+09 y = 3E+07x + 1E+09 0.00E+00 5.00E+08 1.00E+09 1.50E+09 2.00E+09 2.50E+09 3.00E+09 3.50E+09 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 MemoryUsage Time (1s - 67s) Docker / KVM: Serial VM Boot Memory Usage (segment: 1s - 67s) docker kvm Linear (docker) Linear (kvm) 1 second 67 seconds Document v2.0 31. Cloudy Performance: Serial VM Boot 5/11/2014 31 0 5 10 15 20 25 30 35 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 67 69 71 73 75 77 79 1MinuteLoadAverage Time Docker: Compute Node 1m Load Average 1m Average 0.25 % 0 5 10 15 20 25 30 35 1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 70 73 76 79 82 85 88 91 94 97 100 103 106 109 112 115 118 121 124 127 1MinuteLoadAverage Time KVM: Compute Node 1m Load Average 1m Average 11.18 % Document v2.0 32. SERIAL VM SOFT REBOOT OpenStack Cloudy Benchmark 5/11/2014 32Document v2.0 33. Cloudy Performance: Serial VM Reboot - Benchmark scenario overview â Pre-cache VM image on compute node prior to test â Boot a VM & wait for it to become ACTIVE â Soft reboot the VM and wait for it to become ACTIVE â Repeat reboot a total of 5 times â Delete VM â Repeat the above for a total of 5 VMs - Benchmark driver â OpenStack Rally - High level goals â Understand compute node characteristics under sustained VM reboots 5/11/2014 33 0 1 2 3 4 5 6 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 ActiveVMs Time Benchmark Visualization Active VMs Document v2.0 34. Cloudy Performance: Serial VM Reboot 5/11/2014 34 2.577879581 124.433239 0 20 40 60 80 100 120 140 docker KVM TimeInSeconds Average Server Reboot Time docker KVM Document v2.0 35. Cloudy Performance: Serial VM Reboot 5/11/2014 35 3.567586041 3.479760051 0 0.5 1 1.5 2 2.5 3 3.5 4 docker KVM TimeInSeconds Average Server Delete Time docker KVM Document v2.0 36. Cloudy Performance: Serial VM Reboot 5/11/2014 36 0 1 2 3 4 5 6 7 8 9 10 1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 70 73 76 79 82 85 88 91 94 97 100 103 106 109 CPUUsageInPercent Time Docker: Compute Node CPU usr sys 0 1 2 3 4 5 6 7 8 9 10 1 72 143 214 285 356 427 498 569 640 711 782 853 924 995 1066 1137 1208 1279 1350 1421 1492 1563 1634 1705 1776 1847 1918 1989 2060 2131 2202 2273 2344 2415 2486 2557 2628 2699 2770 2841 2912 2983 3054 3125 CPUUsageInPercent Time KVM: Compute Node CPU usr sys Averages â 0.69 â 0.26 Averages â 0.84 â 0.18 Document v2.0 37. Cloudy Performance: Serial VM Reboot 5/11/2014 37 0.00E+00 5.00E+08 1.00E+09 1.50E+09 2.00E+09 2.50E+09 1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 70 73 76 79 82 85 88 91 94 97 100 103 106 109 MemoryUsed Time Docker: Compute Node Used Memory Memory Delta 48 MB 0.00E+00 5.00E+08 1.00E+09 1.50E+09 2.00E+09 2.50E+09 1 81 161 241 321 401 481 561 641 721 801 881 961 1041 1121 1201 1281 1361 1441 1521 1601 1681 1761 1841 1921 2001 2081 2161 2241 2321 2401 2481 2561 2641 2721 2801 2881 2961 3041 3121 MemoryUsed Time KVM: Compute Node Used Memory Memory Delta 486 MB Document v2.0 38. Cloudy Performance: Serial VM Reboot 5/11/2014 38 0 0.5 1 1.5 2 2.5 3 1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 70 73 76 79 82 85 88 91 94 97 100 103 106 109 1MinuteLoadAverage Time Docker: Compute Node 1m Load Average 1m Average 0.4 % 0 0.5 1 1.5 2 2.5 3 1 71 141 211 281 351 421 491 561 631 701 771 841 911 981 1051 1121 1191 1261 1331 1401 1471 1541 1611 1681 1751 1821 1891 1961 2031 2101 2171 2241 2311 2381 2451 2521 2591 2661 2731 2801 2871 2941 3011 3081 3151 1MinuteLoadAverage Time KVM: Compute Node 1m Load Average 1m Average 0.33 % Document v2.0 39. SNAPSHOT VM TO IMAGE OpenStack Cloudy Benchmark 5/11/2014 39Document v2.0 40. Cloudy Performance: Snapshot VM To Image - Benchmark scenario overview â Pre-cache VM image on compute node prior to test â Boot a VM â Wait for it to become ACTIVE â Snapshot the VM â Wait for image to become ACTIVE â Delete VM - Benchmark driver â OpenStack Rally - High level goals â Understand cloudy ops times from a user perspective 5/11/2014 40Document v2.0 41. Cloudy Performance: Snapshot VM To Image 5/11/2014 41 36.88756394 48.02313805 0 10 20 30 40 50 60 docker KVM TimeInSeconds Average Snapshot Server Time docker KVM Document v2.0 42. Cloudy Performance: Snapshot VM To Image 5/11/2014 42 0 1 2 3 4 5 6 7 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 CPUUsageInPercent Time Docker: Compute Node CPU usr sys Averages â 0.42 â 0.15 0 1 2 3 4 5 6 7 1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 70 73 76 79 82 85 88 91 94 97 100 103 106 109 112 115 CPUUsageInPercent Time KVM: Compute Node CPU usr sys Averages â 1.46 â 1.0 Document v2.0 43. Cloudy Performance: Snapshot VM To Image 5/11/2014 43 1.48E+09 1.5E+09 1.52E+09 1.54E+09 1.56E+09 1.58E+09 1.6E+09 1.62E+09 1.64E+09 1.66E+09 1.68E+09 1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 70 73 76 79 82 85 88 91 94 97 100 103 106 109 112 115 MemoryUsed Time KVM: Compute Node Used Memory Memory Delta 114 MB 1.6E+09 1.61E+09 1.62E+09 1.63E+09 1.64E+09 1.65E+09 1.66E+09 1.67E+09 1.68E+09 1.69E+09 1.7E+09 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 MemoryUsed Time Docker: Compute Node Memory Used Memory Delta 57 MB Document v2.0 44. Cloudy Performance: Snapshot VM To Image 5/11/2014 44 0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 1MinuteLoadAverage Time Docker: Compute Node 1m Load Average 1m Average 0.06 % 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 70 73 76 79 82 85 88 91 94 97 100 103 106 109 112 115 1MinuteLoadAverage Time KVM: Compute node 1m Load Average 1m Average 0.47 % Document v2.0 45. GUEST PERFORMANCE BENCHMARKS Guest VM Benchmark 5/11/2014 45Document v2.0 46. Configuring Docker Container for 2CPU x 4G RAM - Configuring docker LXC for 2CPU x 4G RAM â Pin container to 2 CPUs / Mems â Create cpuset cgroup â Pin group to cpuset.mems to 0,1 â Pin group to cpuset.cpus to 0,1 â Add container root proc to tasks â Limit container memory to 4G â Create memory cgroup â Set memory.limit_in_bytes to 4G â Add container root proc to tasks â Limit blkio â Create blkio cgroup â Add container root process of LXC to tasks â Default blkio.weight of 500 5/11/2014 46Document v2.0 47. Guest Performance: CPU - Linux sysbench 0.4.12 cpu test - Calculate prime numbers up to 20000 - 2 threads - Instance size â 4G RAM â 2 CPU cores â 20G disk 5/11/2014 47Document v2.0 48. Guest Performance: CPU 5/11/2014 48 15.26 15.22 15.13 0 2 4 6 8 10 12 14 16 18 Bare Metal docker KVM Seconds Calculate Primes Up To 20000 Bare Metal docker KVM Document v2.0 49. Guest Performance: Memory - Linux mbw 1.1.1-2 - Instance size â 2 CPU â 4G memory - Execution options â 10 runs; average â 1000 MiB 5/11/2014 49Document v2.0 50. Guest Performance: Memory 5/11/2014 50 3823.3 4393.3 12881.61 3813.38 4395.92 12905.68 3428.95 3461.59 7223.23 0 2000 4000 6000 8000 10000 12000 14000 MEMCPY DUMB MCBLOCK MiB/s Memory Test Memory Benchmark Performance Bare Metal (MiB/s) docker (MiB/s) KVM (MiB/s) Document v2.0 51. Guest Performance: Network - Netperf 2.5.0-1 â Netserver running on controller â Netperf on guest â Run netperf 5 times & average results - Instance size â 2 CPU â 4G memory - Execution options â IPv4 / TCP 5/11/2014 51Document v2.0 52. Guest Performance: Network 5/11/2014 52 940.26 940.56 0 100 200 300 400 500 600 700 800 900 1000 docker KVM ThroughputIn10^6bits/second Network Throughput docker KVM Document v2.0 53. Guest Performance: File I/O Random Read - Linux sysbench 0.4.12 fileio test â Synchronous IO â Random read â Total file size of 150G â 16K block size â Test duration of 100s - Thread variations: 1, 8, 16, 32, 64 - Instance size â 4G RAM â 2 CPU cores â 200G disk - KVM specs â Disk cache mode set to none â Virtio â Deadline scheduler (host & guest) - Docker specs â AUFS storage driver â Deadline scheduler 5/11/2014 53Document v2.0 54. Guest Performance: File I/O Random Read 5/11/2014 54 0 500 1000 1500 2000 2500 1 2 4 8 16 32 64 TotalTransferredInKb/sec Threads Sysbench Synchronous File I/O Random Read docker KVM Document v2.0 55. Guest Performance: File I/O Random Read / Write - Linux sysbench 0.4.12 fileio test â Synchronous IO â Random read â Total file size of 150G â 16K block size â Read/Write ratio for combined random IO test: 1.50 â Test duration of 100s - Thread variations: 1, 8, 16, 32, 64 - Instance size â 4G RAM â 2 CPU cores â 200G disk - KVM specs â Disk cache mode set to none â Virtio â Deadline scheduler (host & guest) - Docker specs â AUFS storage driver â Deadline scheduler 5/11/2014 55Document v2.0 56. Guest Performance: File I/O Random Read / Write 5/11/2014 56 0 200 400 600 800 1000 1200 1400 1600 1 2 4 8 16 32 64 TotalTransferredInKb/sec Threads Sysbench Synchronous File I/O Random Read/Write @ R/W Ratio of 1.50 docker KVM Document v2.0 57. Guest Performance: MySQL OLTP - Linux sysbench 0.4.12 oltp test â Table size of 2,000,000 â MySQL 5.5 (installed on Ubuntu 12.04 LTS with apt-get) â 60 second iterations â Default MySQL cnf settings - Variations â Number of threads â Transactional random read & transactional random read / write - Instance size â 4G RAM â 2 CPU cores â 20G disk 5/11/2014 57Document v2.0 58. Guest Performance: MySQL OLTP 5/11/2014 58 0 5000 10000 15000 20000 25000 30000 35000 40000 45000 50000 1 2 4 8 16 32 64 TotalTransactions Threads MySQL OLTP Random Transactional Reads (60s) docker KVM Document v2.0 59. Guest Performance: MySQL OLTP 5/11/2014 59 0 2000 4000 6000 8000 10000 12000 14000 1 2 4 8 16 32 64 TotalTransactions Threads MySQL OLTP Random Transactional R/W (60s) docker KVM Document v2.0 60. Guest Performance: MySQL Indexed Insertion - Indexed insertion benchmark (iibench python script) â A total of 1,000,000 insertions â Print stats at 100K intervals - Instance size â 4G RAM â 2 CPU cores â 20G disk 5/11/2014 60Document v2.0 61. Guest Performance: MySQL Indexed Insertion 5/11/2014 61 0 20 40 60 80 100 120 140 100000 200000 300000 400000 500000 600000 700000 800000 900000 1000000 SecondsPer100KInsertionBatch Table Size In Rows MySQL Indexed Insertion @ 100K Intervals docker kvm Document v2.0 62. Guest Performance: BlogBench - Blogbench 1.1 â Test duration of 5m â Average results over 5 iterations of test - Instance size â 4G RAM â 2 CPU cores â 200G disk - KVM specs â Disk cache mode set to none â Virtio â Deadline scheduler (host & guest) - Docker specs â AUFS storage driver â Deadline scheduler 5/11/2014 62Document v2.0 63. Guest Performance: BlogBench 5/11/2014 63 398772.6 384769 0 50000 100000 150000 200000 250000 300000 350000 400000 450000 docker KVM Score Blogbench Read Scores docker KVM 1526.6 1285 0 200 400 600 800 1000 1200 1400 1600 1800 docker KVM Score Blogbench Write Scores docker KVM Document v2.0 64. OTHER CONSIDERATIONS 5/11/2014 64Document v2.0 65. Cloud Management Impacts on LXC 5/11/2014 65 0.17 3.529113102 0 0.5 1 1.5 2 2.5 3 3.5 4 docker cli nova-docker Seconds Docker: Boot Container - CLI vs Nova Virt docker cli nova-docker Cloud management often caps true ops performance of LXC Document v2.0 66. Ubuntu MySQL Image Size 5/11/2014 Document v2.0 66 381.5 1080 0 200 400 600 800 1000 1200 docker kvm SizeInMB Docker / KVM: Ubuntu MySQL docker kvm Out of the box JeOS images for docker are lightweight 67. Other Observations - Micro âsyntheticâ benchmarks do not reflect macro âapplicationâ performance â Always benchmark your ârealâ workload - Nova-docker virt driver still under development â Great start, but additional features needed for parity (python anyone?) â Additions to the nova-docker driver could change Cloudy performance - Docker LXC is still under development â Docker has not yet released v1.0 for production readiness - KVM images can be made skinnier, but requires additional effort - Increased density / oversubscription imposes additional complexity â Techniques to handle resource consumption surges which exceed capacity 5/11/2014 Document v2.0 67 68. REFERENCE 5/11/2014 68Document v2.0 69. References & Related Links - http://www.slideshare.net/BodenRussell/realizing-linux-containerslxc - http://www.slideshare.net/BodenRussell/kvm-and-docker-lxc-benchmarking-with- openstack - https://github.com/bodenr/cloudy-docker-kvm-bench - https://www.docker.io/ - http://sysbench.sourceforge.net/ - http://dag.wiee.rs/home-made/dstat/ - http://www.openstack.org/ - https://wiki.openstack.org/wiki/Rally - https://wiki.openstack.org/wiki/Docker - http://devstack.org/ - http://www.linux-kvm.org/page/Main_Page - https://github.com/stackforge/nova-docker - https://github.com/dotcloud/docker-registry - http://www.netperf.org/netperf/ - http://www.tokutek.com/products/iibench/ - http://www.brendangregg.com/activebenchmarking.html 5/11/2014 69Document v2.0 70. Cloudy Benchmark: Serially Boot 15 VMs - KVM +------------------+-------+---------------+---------------+---------------+---------------+---------------+ | action | count | max (sec) | avg (sec) | min (sec) | 90 percentile | 95 percentile | +------------------+-------+---------------+---------------+---------------+---------------+---------------+ | nova.boot_server | 15 | 7.37148094177 | 5.78166244825 | 4.77369403839 | 6.67956886292 | 7.07061390877 | +------------------+-------+---------------+---------------+---------------+---------------+---------------+ +---------------+---------------+---------------+---------------+---------------+---------------+-------------+ | max (sec) | avg (sec) | min (sec) | 90 pecentile | 95 percentile | success/total | total times | +---------------+---------------+---------------+---------------+---------------+---------------+-------------+ | 7.58968496323 | 6.00853565534 | 4.99443006516 | 6.91288709641 | 7.28662061691 | 1.0 | 15 | +---------------+---------------+---------------+---------------+---------------+---------------+-------------+ - Docker +------------------+-------+---------------+---------------+---------------+---------------+---------------+ | action | count | max (sec) | avg (sec) | min (sec) | 90 percentile | 95 percentile | +------------------+-------+---------------+---------------+---------------+---------------+---------------+ | nova.boot_server | 15 | 5.18499684334 | 3.52911310196 | 2.93864893913 | 4.74490590096 | 4.95752367973 | +------------------+-------+---------------+---------------+---------------+---------------+---------------+ +---------------+---------------+---------------+---------------+---------------+---------------+-------------+ | max (sec) | avg (sec) | min (sec) | 90 pecentile | 95 percentile | success/total | total times | +---------------+---------------+---------------+---------------+---------------+---------------+-------------+ | 5.43275094032 | 3.77053097089 | 3.12985610962 | 4.95886874199 | 5.18047580719 | 1.0 | 15 | +---------------+---------------+---------------+---------------+---------------+---------------+-------------+ 5/11/2014 70Document v2.0 71. Cloudy Performance: Serial VM Reboot - KVM +--------------------+-------+---------------+---------------+---------------+---------------+---------------+ | action | count | max (sec) | avg (sec) | min (sec) | 90 percentile | 95 percentile | +--------------------+-------+---------------+---------------+---------------+---------------+---------------+ | nova.reboot_server | 10 | 124.900292158 | 124.433238959 | 123.947879076 | 124.881286669 | 124.890789413 | | nova.boot_server | 2 | 7.05096197128 | 6.82815694809 | 6.6053519249 | 7.00640096664 | 7.02868146896 | | nova.delete_server | 2 | 4.46658396721 | 3.47976005077 | 2.49293613434 | 4.26921918392 | 4.36790157557 | +--------------------+-------+---------------+---------------+---------------+---------------+---------------+ +---------------+---------------+---------------+---------------+---------------+---------------+-------------+ | max (sec) | avg (sec) | min (sec) | 90 pecentile | 95 percentile | success/total | total times | +---------------+---------------+---------------+---------------+---------------+---------------+-------------+ | 633.087348938 | 632.493344903 | 631.899340868 | 632.968548131 | 633.027948534 | 0.4 | 5 | +---------------+---------------+---------------+---------------+---------------+---------------+-------------+ - Docker +--------------------+-------+---------------+---------------+---------------+---------------+---------------+ | action | count | max (sec) | avg (sec) | min (sec) | 90 percentile | 95 percentile | +--------------------+-------+---------------+---------------+---------------+---------------+---------------+ | nova.reboot_server | 25 | 4.48567795753 | 2.57787958145 | 2.35410904884 | 3.0847319603 | 3.48342533112 | | nova.boot_server | 5 | 4.16244912148 | 3.5675860405 | 3.05103397369 | 4.03664107323 | 4.09954509735 | | nova.delete_server | 5 | 3.54331803322 | 3.52483625412 | 3.50456190109 | 3.53761086464 | 3.54046444893 | +--------------------+-------+---------------+---------------+---------------+---------------+---------------+ +---------------+---------------+---------------+--------------+---------------+---------------+-------------+ | max (sec) | avg (sec) | min (sec) | 90 pecentile | 95 percentile | success/total | total times | +---------------+---------------+---------------+--------------+---------------+---------------+-------------+ | 21.5702910423 | 19.9976443768 | 18.7037060261 | 20.997631073 | 21.2839610577 | 1.0 | 5 | +---------------+---------------+---------------+--------------+---------------+---------------+-------------+ 5/11/2014 71Document v2.0 72. Cloud Performance: Snapshot VM To Image - KVM +--------------------+-------+----------------+----------------+----------------+----------------+----------------+ | action | count | max (sec) | avg (sec) | min (sec) | 90 percentile | 95 percentile | +--------------------+-------+----------------+----------------+----------------+----------------+----------------+ | nova.delete_image | 1 | 0.726859092712 | 0.726859092712 | 0.726859092712 | 0.726859092712 | 0.726859092712 | | nova.create_image | 1 | 48.0231380463 | 48.0231380463 | 48.0231380463 | 48.0231380463 | 48.0231380463 | | nova.boot_server | 2 | 32.7824101448 | 19.4164011478 | 6.05039215088 | 30.1092083454 | 31.4458092451 | | nova.delete_server | 2 | 12.3564949036 | 8.40917897224 | 4.46186304092 | 11.5670317173 | 11.9617633104 | +--------------------+-------+----------------+----------------+----------------+----------------+----------------+ +---------------+---------------+---------------+---------------+---------------+---------------+-------------+ | max (sec) | avg (sec) | min (sec) | 90 pecentile | 95 percentile | success/total | total times | +---------------+---------------+---------------+---------------+---------------+---------------+-------------+ | 104.401446104 | 104.401446104 | 104.401446104 | 104.401446104 | 104.401446104 | 1.0 | 1 | +---------------+---------------+---------------+---------------+---------------+---------------+-------------+ - Docker (defect deleting image) +--------------------+-------+---------------+---------------+---------------+---------------+---------------+ | action | count | max (sec) | avg (sec) | min (sec) | 90 percentile | 95 percentile | +--------------------+-------+---------------+---------------+---------------+---------------+---------------+ | nova.create_image | 1 | 36.8875639439 | 36.8875639439 | 36.8875639439 | 36.8875639439 | 36.8875639439 | | nova.boot_server | 2 | 3.96964478493 | 3.84809792042 | 3.72655105591 | 3.94533541203 | 3.95749009848 | | nova.delete_server | 2 | 4.48610281944 | 4.46519696712 | 4.44429111481 | 4.48192164898 | 4.48401223421 | +--------------------+-------+---------------+---------------+---------------+---------------+---------------+ +-----------+-----------+-----------+--------------+---------------+---------------+-------------+ | max (sec) | avg (sec) | min (sec) | 90 pecentile | 95 percentile | success/total | total times | +-----------+-----------+-----------+--------------+---------------+---------------+-------------+ | n/a | n/a | n/a | n/a | n/a | 0 | 1 | +-----------+-----------+-----------+--------------+---------------+---------------+-------------+ 5/11/2014 72Document v2.0 https://www.friendbookmark.com/videos/964/infrastructure-as-code-running-microservices-on-aws-using-docker-terraform-and-ecsThis is a talk about managing your software and infrastructure-as-code that walks through a real-world example of deploying microservices on AWS using Docker, Terraform, and ECS. Topics covered in this presentation slides: 1. INFRASTRUCTURE as CODE Running Microservices on AWS with Docker, Terraform, and ECS 2. Why infrastructure-as-code matters: a short story. 3. You are starting a new project 4. I know, Iâll use Ruby on Rails! 5. > gem install rails 6. > gem install rails Fetching: i18n-0.7.0.gem (100%) Fetching: json-1.8.3.gem (100%) Building native extensions. This could take a while... ERROR: Error installing rails: ERROR: Failed to build gem native extension. /usr/bin/ruby1.9.1 extconf.rb creating Makefile make sh: 1: make: not found 7. Ah, I just need to install make 8. > sudo apt-get install make ... Success! 9. > gem install rails 10. > gem install rails Fetching: nokogiri-1.6.7.2.gem (100%) Building native extensions. This could take a while... ERROR: Error installing rails: ERROR: Failed to build gem native extension. /usr/bin/ruby1.9.1 extconf.rb checking if the C compiler accepts ... yes Building nokogiri using packaged libraries. Using mini_portile version 2.0.0.rc2 checking for gzdopen() in -lz... no zlib is missing; necessary for building libxml2 *** extconf.rb failed *** 11. Hmm. Time to visit StackOverflow. 12. > sudo apt-get install zlib1g-dev ... Success! 13. > gem install rails 14. > gem install rails Building native extensions. This could take a while... ERROR: Error installing rails: ERROR: Failed to build gem native extension. /usr/bin/ruby1.9.1 extconf.rb checking if the C compiler accepts ... yes Building nokogiri using packaged libraries. Using mini_portile version 2.0.0.rc2 checking for gzdopen() in -lz... yes checking for iconv... yes Extracting libxml2-2.9.2.tar.gz into tmp/x86_64-pc-linux- gnu/ports/libxml2/2.9.2... OK *** extconf.rb failed *** 15. nokogiri y u never install correctly? 16. (Spend 2 hours trying random StackOverflow suggestions) 17. > gem install rails 18. > gem install rails ... Success! 19. Finally! 20. > rails new my-project > cd my-project > rails start 21. > rails new my-project > cd my-project > rails start /source/my-project/bin/spring:11:in `': undefined method `path_separator' for Gem:Module (NoMethodError) from bin/rails:3:in `load' from bin/rails:3:in `' 22. Eventually, you get it working 23. Now you have to deploy your Rails app in production 24. You use the AWS Console to deploy an EC2 instance 25. > ssh ec2-user@ec2-12-34-56-78.compute-1.amazonaws.com __| __|_ ) _| ( / Amazon Linux AMI ___|___|___| [ec2-user@ip-172-31-61-204 ~]$ gem install rails 26. > ssh ec2-user@ec2-12-34-56-78.compute-1.amazonaws.com __| __|_ ) _| ( / Amazon Linux AMI ___|___|___| [ec2-user@ip-172-31-61-204 ~]$ gem install rails ERROR: Error installing rails: ERROR: Failed to build gem native extension. /usr/bin/ruby1.9.1 extconf.rb 27. Eventually you get it working 28. Now you urgently have to update all your Rails installs 29. > bundle update rails 30. > bundle update rails Building native extensions. This could take a while... ERROR: Error installing rails: ERROR: Failed to build gem native extension. /usr/bin/ruby1.9.1 extconf.rb checking if the C compiler accepts ... yes Building nokogiri using packaged libraries. Using mini_portile version 2.0.0.rc2 checking for gzdopen() in -lz... yes checking for iconv... yes Extracting libxml2-2.9.2.tar.gz into tmp/x86_64-pc-linux- gnu/ports/libxml2/2.9.2... OK *** extconf.rb failed *** 31. The problem isnât Rails 32. > ssh ec2-user@ec2-12-34-56-78.compute-1.amazonaws.com __| __|_ ) _| ( / Amazon Linux AMI ___|___|___| [ec2-user@ip-172-31-61-204 ~]$ gem install rails The problem is that youâre configuring servers manually 33. And that youâre deploying infrastructure manually 34. A better alternative: infrastructure- as-code 35. In this talk, weâll go through a real-world example: 36. Weâll configure & deploy two microservices on Amazon ECS 37. With two infrastructure-as-code tools: Docker and Terraform TERRAFORM 38. Iâm Yevgeniy Brikman ybrikman.com 39. Co-founder of Gruntwork gruntwork.io 40. gruntwork.io We offer DevOps as a Service 41. gruntwork.io And DevOps as a Library 42. PAST LIVES 43. Author of Hello, Startup hello-startup.net 44. And Terraform: Up & Running terraformupandrunning.com 45. Slides and code from this talk: ybrikman.com/speaking 46. 1. Microservices 2. Docker 3. Terraform 4. ECS 5. Recap Outline 47. 1. Microservices 2. Docker 3. Terraform 4. ECS 5. Recap Outline 48. Code is the enemy: the more you have, the slower you go 49. Project Size Lines of code Bug Density Bugs per thousand lines of code < 2K 0 â 25 2K â 6K 0 â 40 16K â 64K 0.5 â 50 64K â 512K 2 â 70 > 512K 4 â 100 50. As the code grows, the number of bugs grows even faster 51. âSoftware development doesn't happen in a chart, an IDE, or a design tool; it happens in your head.â 52. The mind can only handle so much complexity at once 53. One solution is to break the code into microservices 54. In a monolith, you use function calls within one process moduleA.func() moduleB.func() moduleC.func() moduleD.func() moduleE.func() 55. http://service.a http://service.b http://service.c http://service.d http://service.e With services, you pass messages between processes 56. Advantages of services: 1. Isolation 2. Technology agnostic 3. Scalability 57. Disadvantages of services: 1. Operational overhead 2. Performance overhead 3. I/O, error handling 4. Backwards compatibility 5. Global changes, transactions, referential integrity all very hard 58. For more info, see: Splitting Up a Codebase into Microservices and Artifacts 59. For this talk, weâll use two example microservices 60. require 'sinatra' get "/" do "Hello, World!" end A sinatra backend that returns âHello, Worldâ 61. class ApplicationController < ActionController::Base def index url = URI.parse(backend_addr) req = Net::HTTP::Get.new(url.to_s) res = Net::HTTP.start(url.host, url.port) {|http| http.request(req) } @text = res.body end end A rails frontend that calls the sinatra backend 62. Rails Frontend Response from the backend: And renders the response as HTML 63. 1. Microservices 2. Docker 3. Terraform 4. ECS 5. Recap Outline 64. Docker allows you to build and run code in containers 65. Containers are like lightweight Virtual Machines (VMs) 66. VMs virtualize the hardware and run an entire guest OS on top of the host OS VM Hardware Host OS Host User Space Virtual Machine Virtualized hardware Guest OS Guest User Space App VM Virtualized hardware Guest OS Guest User Space App VM Virtualized hardware Guest OS Guest User Space App 67. This provides good isolation, but lots of CPU, memory, disk, & startup overhead VM Hardware Host OS Host User Space Virtual Machine Virtualized hardware Guest OS Guest User Space App VM Virtualized hardware Guest OS Guest User Space App VM Virtualized hardware Guest OS Guest User Space App 68. Containers virtualize User Space (shared memory, processes, mount, network) Container VM Hardware Host OS Host User Space Virtual Machine Virtualized hardware Guest OS Guest User Space App Hardware Host OS Host User Space Container Engine Virtualized User Space VM Virtualized hardware Guest OS Guest User Space App VM Virtualized hardware Guest OS Guest User Space App App Container Virtualized User Space App Container Virtualized User Space App 69. Container VM Hardware Host OS Host User Space Virtual Machine Virtualized hardware Guest OS Guest User Space App Hardware Host OS Host User Space Container Engine Virtualized User Space VM Virtualized hardware Guest OS Guest User Space App VM Virtualized hardware Guest OS Guest User Space App App Container Virtualized User Space App Container Virtualized User Space App Isolation isnât as good but much less CPU, memory, disk, startup overhead 70. > docker run âit ubuntu bash root@12345:/# echo "I'm in $(cat /etc/issue)â I'm in Ubuntu 14.04.4 LTS Running Ubuntu in a Docker container 71. > time docker run ubuntu echo "Hello, World" Hello, World real 0m0.183s user 0m0.009s sys 0m0.014s Containers boot very quickly. Easily run a dozen at once. 72. You can define a Docker image as code in a Dockerfile 73. FROM gliderlabs/alpine:3.3 RUN apk --no-cache add ruby ruby-dev RUN gem install sinatra --no-ri --no-rdoc RUN mkdir -p /usr/src/app COPY . /usr/src/app WORKDIR /usr/src/app EXPOSE 4567 CMD ["ruby", "app.rb"] Here is the Dockerfile for the Sinatra backend 74. FROM gliderlabs/alpine:3.3 RUN apk --no-cache add ruby ruby-dev RUN gem install sinatra --no-ri --no-rdoc RUN mkdir -p /usr/src/app COPY . /usr/src/app WORKDIR /usr/src/app EXPOSE 4567 CMD ["ruby", "app.rb"] It specifies dependencies, code, config, and how to run the app 75. > docker build -t brikis98/sinatra-backend . Step 0 : FROM gliderlabs/alpine:3.3 ---> 0a7e169bce21 (...) Step 8 : CMD ruby app.rb ---> 2e243eba30ed Successfully built 2e243eba30ed Build the Docker image 76. > docker run -it -p 4567:4567 brikis98/sinatra-backend INFO WEBrick 1.3.1 INFO ruby 2.2.4 (2015-12-16) [x86_64-linux-musl] == Sinatra (v1.4.7) has taken the stage on 4567 for development with backup from WEBrick INFO WEBrick::HTTPServer#start: pid=1 port=4567 Run the Docker image 77. > docker push brikis98/sinatra-backend The push refers to a repository [docker.io/brikis98/sinatra- backend] (len: 1) 2e243eba30ed: Image successfully pushed 7e2e0c53e246: Image successfully pushed 919d9a73b500: Image successfully pushed (...) v1: digest: sha256:09f48ed773966ec7fe4558 size: 14319 You can share your images by pushing them to Docker Hub 78. Now you can reuse the same image in dev, stg, prod, etc 79. > docker pull rails:4.2.6 And you can reuse images created by others. 80. FROM rails:4.2.6 RUN mkdir -p /usr/src/app COPY . /usr/src/app WORKDIR /usr/src/app RUN bundle install EXPOSE 3000 CMD ["rails", "start"] The rails-frontend is built on top of the official rails Docker image 81. No more insane install procedures! 82. rails_frontend: image: brikis98/rails-frontend ports: - "3000:3000" links: - sinatra_backend:sinatra_backend sinatra_backend: image: brikis98/sinatra-backend ports: - "4567:4567" Define your entire dev stack as code with docker-compose 83. rails_frontend: image: brikis98/rails-frontend ports: - "3000:3000" links: - sinatra_backend sinatra_backend: image: brikis98/sinatra-backend ports: - "4567:4567" Docker links provide a simple service discovery mechanism 84. > docker-compose up Starting infrastructureascodetalk_sinatra_backend_1 Recreating infrastructureascodetalk_rails_frontend_1 sinatra_backend_1 | INFO WEBrick 1.3.1 sinatra_backend_1 | INFO ruby 2.2.4 (2015-12-16) sinatra_backend_1 | Sinatra has taken the stage on 4567 rails_frontend_1 | INFO WEBrick 1.3.1 rails_frontend_1 | INFO ruby 2.3.0 (2015-12-25) rails_frontend_1 | INFO WEBrick::HTTPServer#start: port=3000 Run your entire dev stack with one command 85. Advantages of Docker: 1. Easy to create & share images 2. Images run the same way in all environments (dev, test, prod) 3. Easily run the entire stack in dev 4. Minimal overhead 5. Better resource utilization 86. Disadvantages of Docker: 1. Maturity. Ecosystem developing very fast, but still a ways to go 2. Tricky to manage persistent data in a container 3. Tricky to pass secrets to containers 87. 1. Microservices 2. Docker 3. Terraform 4. ECS 5. Recap Outline 88. Terraform is a tool for provisioning infrastructure 89. Terraform supports many providers (cloud agnostic) 90. And many resources for each provider 91. You define infrastructure as code in Terraform templates 92. provider "aws" { region = "us-east-1" } resource "aws_instance" "example" { ami = "ami-408c7f28" instance_type = "t2.micro" } This template creates a single EC2 instance in AWS 93. > terraform plan + aws_instance.example ami: "" => "ami-408c7f28" instance_type: "" => "t2.micro" key_name: "" => "" private_ip: "" => "" public_ip: "" => "" Plan: 1 to add, 0 to change, 0 to destroy. Use the plan command to see what youâre about to deploy 94. > terraform apply aws_instance.example: Creating... ami: "" => "ami-408c7f28" instance_type: "" => "t2.micro" key_name: "" => "" private_ip: "" => "" public_ip: "" => "â aws_instance.example: Creation complete Apply complete! Resources: 1 added, 0 changed, 0 destroyed. Use the apply command to apply the changes 95. Now our EC2 instance is running! 96. resource "aws_instance" "example" { ami = "ami-408c7f28" instance_type = "t2.micro" tags { Name = "terraform-example" } } Letâs give the EC2 instance a tag with a readable name 97. > terraform plan ~ aws_instance.example tags.#: "0" => "1" tags.Name: "" => "terraform-example" Plan: 0 to add, 1 to change, 0 to destroy. Use the plan command again to verify your changes 98. > terraform apply aws_instance.example: Refreshing state... aws_instance.example: Modifying... tags.#: "0" => "1" tags.Name: "" => "terraform-example" aws_instance.example: Modifications complete Apply complete! Resources: 0 added, 1 changed, 0 destroyed. Use the apply command again to deploy those changes 99. Now our EC2 instance has a tag! 100. resource "aws_elb" "example" { name = "example" availability_zones = ["us-east-1a", "us-east-1b"] instances = ["${aws_instance.example.id}"] listener { lb_port = 80 lb_protocol = "http" instance_port = "${var.instance_port}" instance_protocol = "httpâ } } Letâs add an Elastic Load Balancer (ELB). 101. resource "aws_elb" "example" { name = "example" availability_zones = ["us-east-1a", "us-east-1b"] instances = ["${aws_instance.example.id}"] listener { lb_port = 80 lb_protocol = "http" instance_port = "${var.instance_port}" instance_protocol = "httpâ } } Terraform supports variables, such as var.instance_port 102. resource "aws_elb" "example" { name = "example" availability_zones = ["us-east-1a", "us-east-1b"] instances = ["${aws_instance.example.id}"] listener { lb_port = 80 lb_protocol = "http" instance_port = "${var.instance_port}" instance_protocol = "http" } } As well as dependencies like aws_instance.example.id 103. resource "aws_elb" "example" { name = "example" availability_zones = ["us-east-1a", "us-east-1b"] instances = ["${aws_instance.example.id}"] listener { lb_port = 80 lb_protocol = "http" instance_port = "${var.instance_port}" instance_protocol = "http" } } It builds a dependency graph and applies it in parallel. 104. After running apply, we have an ELB! 105. > terraform destroy aws_instance.example: Refreshing state... (ID: i-f3d58c70) aws_elb.example: Refreshing state... (ID: example) aws_elb.example: Destroying... aws_elb.example: Destruction complete aws_instance.example: Destroying... aws_instance.example: Destruction complete Apply complete! Resources: 0 added, 0 changed, 2 destroyed. Use the destroy command to delete all your resources 106. For more info, check out The Comprehensive Guide to Terraform 107. Advantages of Terraform: 1. Concise, readable syntax 2. Reusable code: inputs, outputs, modules 3. Plan command! 4. Cloud agnostic 5. Very active development 108. Disadvantages of Terraform: 1. Maturity 2. Collaboration on Terraform state is hard (but terragrunt makes it easier) 3. No rollback 4. Poor secrets management 109. 1. Microservices 2. Docker 3. Terraform 4. ECS 5. Recap Outline 110. EC2 Container Service (ECS) is a way to run Docker on AWS 111. ECS Overview EC2 Instance ECS Cluster ECS Scheduler ECS Agent ECS Tasks ECS Task Definition { "cluster": "example", "serviceName": âfoo", "taskDefinition": "", "desiredCount": 2 } ECS Service Definition { "name": "example", "image": "foo/example", "cpu": 1024, "memory": 2048, "essential": true, } 112. ECS Cluster: several servers managed by ECS EC2 Instance ECS Cluster 113. Typically, the servers are in an Auto Scaling Group EC2 Instance Auto Scaling Group 114. Which can automatically relaunch failed servers EC2 Instance Auto Scaling Group 115. Each server must run the ECS Agent EC2 Instance ECS Cluster ECS Agent 116. ECS Task: Docker container(s) to run, resources they need EC2 Instance ECS Cluster ECS Agent ECS Task Definition { "name": "example", "image": "foo/example", "cpu": 1024, "memory": 2048, "essential": true, } 117. ECS Service: long-running ECS Task & ELB settings EC2 Instance ECS Cluster ECS Agent ECS Task Definition { "name": "example", "image": "foo/example", "cpu": 1024, "memory": 2048, "essential": true, } { "cluster": "example", "serviceName": âfoo", "taskDefinition": "", "desiredCount": 2 } ECS Service Definition 118. ECS Scheduler: Deploys Tasks across the ECS Cluster EC2 Instance ECS Cluster ECS Agent ECS Task Definition { "name": "example", "image": "foo/example", "cpu": 1024, "memory": 2048, "essential": true, } { "cluster": "example", "serviceName": âfoo", "taskDefinition": "", "desiredCount": 2 } ECS Service Definition ECS Scheduler ECS Tasks 119. It will also automatically redeploy failed Services EC2 Instance ECS Cluster ECS Agent ECS Task Definition { "name": "example", "image": "foo/example", "cpu": 1024, "memory": 2048, "essential": true, } { "cluster": "example", "serviceName": âfoo", "taskDefinition": "", "desiredCount": 2 } ECS Service Definition ECS Scheduler ECS Tasks 120. You can associate an ALB or ELB with each ECS service EC2 Instance ECS Cluster ECS Agent ECS Tasks User 121. This lets you distribute traffic across multiple ECS Tasks EC2 Instance ECS Cluster ECS Agent ECS Tasks User 122. Which allows zero-downtime deployment EC2 Instance ECS Cluster ECS Agent ECS TasksUser v1 v1 v1 v2 123. As well as a simple form of service discovery EC2 Instance ECS Cluster ECS Agent ECS Tasks 124. You can use CloudWatch alarms to trigger auto scaling EC2 Instance ECS Cluster ECS Agent ECS Tasks CloudWatch 125. You can scale up by running more ECS Tasks EC2 Instance ECS Cluster ECS Agent ECS Tasks CloudWatch 126. And by adding more EC2 Instances EC2 Instance ECS Cluster ECS Agent ECS Tasks CloudWatch 127. And scale back down when load is lower EC2 Instance ECS Cluster ECS Agent ECS Tasks CloudWatch 128. Letâs deploy our microservices in ECS using Terraform 129. Define the ECS Cluster as an Auto Scaling Group (ASG) EC2 Instance ECS Cluster 130. resource "aws_ecs_cluster" "example_cluster" { name = "example-cluster" } resource "aws_autoscaling_group" "ecs_cluster_instances" { name = "ecs-cluster-instances" min_size = 3 max_size = 3 launch_configuration = "${aws_launch_configuration.ecs_instance.name}" } 131. Ensure each server in the ASG runs the ECS Agent EC2 Instance ECS Cluster ECS Agent 132. # The launch config defines what runs on each EC2 instance resource "aws_launch_configuration" "ecs_instance" { name_prefix = "ecs-instance-" instance_type = "t2.micro" # This is an Amazon ECS AMI, which has an ECS Agent # installed that lets it talk to the ECS cluster image_id = "ami-a98cb2c3â } The launch config runs AWS ECS Linux on each server in the ASG 133. Define an ECS Task for each microservice EC2 Instance ECS Cluster ECS Agent ECS Task Definition { "name": "example", "image": "foo/example", "cpu": 1024, "memory": 2048, "essential": true, } 134. resource "aws_ecs_task_definition" "rails_frontend" { family = "rails-frontend" container_definitions = "" container_definitions: "bb5352f" => "2ff6ae" (forces new resource) revision: "3" => "â Plan: 1 to add, 1 to change, 1 to destroy. Use the plan command to verify the changes 154. Apply the changes and youâll see v2. 155. Advantages of ECS: 1. One of the simplest Docker cluster management tools 2. Almost no extra cost if on AWS 3. Pluggable scheduler 4. Auto-restart of instances & Tasks 5. Automatic ALB/ELB integration 156. Disadvantages of ECS: 1. UI is so-so 2. Minimal monitoring built-in 3. ALB is broken 157. 1. Microservices 2. Docker 3. Terraform 4. ECS 5. Recap Outline 158. Benefits of infrastructure-as-code: 1. Reuse 2. Automation 3. Version control 4. Code review 5. Testing 6. Documentation 159. Slides and code from this talk: ybrikman.com/speaking 160. For more info, see Hello, Startup hello-startup.net 161. And Terraform: Up & Running terraformupandrunning.com 162. gruntwork.io For DevOps help, see Gruntwork 163. Questions? https://www.friendbookmark.com/videos/963/docker-fundamentalsWhat is Docker and Docker Applications. 1. Docker Fundamentals Alper UNAL 2. Content - Understanding the DevOps - The Docker Technology - Install Docker Server - Docker Machine - Docker Commands - Docker Registry and Repositories - Creating and Managing Docker Images - Running and Managing Containers - Creating and Running a Simple Web App. - GitHub - Docker Networking Basics - Docker Compose - YAML files - Scaling out with Swarm - What is next? â Kubernetes â Openshift â CI/CD Servers â Ansible / Puppet / Chef 3. 1. Introduction - What is Docker? â In 2013, started as opensource project at dotCloud,Inc. â Renamed as Docker,Inc. at October, 2013 - Infrastructure Shifts - 90s Pre-Virtualization: Physical Servers (80s:Mainframes) Problems: â Huge Cost â Slow Deployment â Hard to Migrate 4. Hypervisor Virtualization - 2000s Hypervisor Virtualization: VMWare, HyperV, Logical Domains Benefits: â Cost-Efficient â Easy to Scale Limitations: â Resource Duplication â Application Portability 5. Cloud - 2010s Cloud Technologies â Amazon Web Services, Microsoft Azure and Google Cloud Platform, IBM with 34b$ Acqusition of Red Hat Amazon's Flagship flagship AWS Lambda launched in 2014. Lambda can be triggered by AWS services such as Amazon Simple Storage Service (S3), DynamoDB, Kinesis, SNS, and CloudWatch Google App Engine launched 2008. App Engine supports Node.js, Java, Ruby, C#, Go, Python, and PHP and database products are Cloud Datastore and Firebase. Kubernetes was created by Google in 2015 and is an open-source platform  Flagship, Azure Functions, allows users users to execute their code, written in languages including JavaScript, C#. Functions also interact with other Azure products including Azure Cosmos DB and Azure Storage. 6. Container Virtualization - 2015s: Container Technologies Benefits: â Cost-Efficient â Fast Deployment â Portability 7. Hypervisor vs. Container Virtualization 8. DevOps - DevOps is an IT mindset that encourages communication, collaboration, integration and automation among software developers and IT operations in order to improve the speed and quality of delivering software - DevOps is the offspring of agile software development - DevOps Practices: â Continuous Integration â Continuous Delivery â Microservices â Infrastructure as Code â Monitoring and Logging â Communication and Collaboration 9. 2. The Docker Technology - Docker Client â Server Architecture â Docker Server Docker Daemon running on Docker Host Also referred as Docker Engine â Docker Client CLI: $ docker build/pull/run GUI: Kitematic - Docker Fastest Growing Cloud Tech - By 2020 %50 of global orgs use Docker - Docker Hub Pulls: 2014:1M, 2015:1B, 2016:6B, 2017:24B 10. Docker Architecture 11. Docker on Linux and OSX 12. Docker on Windows 13. Docker on Windows - Docker and Microsoft Bring Containers to Windows Apps - All Windows Server 2016 and later versions come with Docker Engine - Enterprise. Additionally, developers can leverage Docker natively with Windows 10 via Docker Desktop (Development Environment) 14. Docker Machine - Docker Machine is a tool for provisioning and managing your Dockerized hosts (hosts with Docker Engine on them). - Typically, you install Docker Machine on your local system. Docker Machine has its own command line client docker- machine and the Docker Engine client, docker. - You can use Machine to install Docker Engine on one or more virtual systems. These virtual systems can be local (as when you use Machine to install and run Docker Engine in VirtualBox on Mac or Windows) or remote (as when you use Machine to provision Dockerized hosts on cloud providers). - The Dockerized hosts themselves can be thought of, and are sometimes referred to as, managed âmachinesâ. 15. Docker Machine 16. Docker EE - Docker Enterprise 2.1 is a Containers-as-a-Service (CaaS) - The default Docker Enterprise installation includes both Kubernetes and Swarm components across the cluster 17. Docker EE vs. CE 18. 3. Installation - How to Install Docker for Windows â https://docs.docker.com/docker-for-windows/install/ - Docker for Windows requires Microsoft Hyper-V to run â The Docker for Windows installer enables Hyper-V â You need Windows 10 or Windows Server 2016 to install Docker for Windows. This is preferred since it runs as native app, but you can not use VirtualBox images anymore â If your system does not meet the requirements to run Docker for Windows, you can install Docker Toolbox, which uses Oracle Virtual Box instead of Hyper-V â Docker for Windows install includes: Docker Engine, Docker CLI client, Docker Compose, Docker Machine, and Kitematic. â After installation check: DockerQuickstartTerminal and Kitematic app. 19. Install Docker on Linux - Ubuntu18 â https://docs.docker.com/install/linux/docker-ce/ubuntu/ - Postinstall Tasks for Linux: Create a Docker User â https://docs.docker.com/install/linux/linux-postinstall/ - CentOS7 â https://docs.docker.com/install/linux/docker-ce/centos/ â Add Docker repo and Install Docker CE sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo sudo yum install docker-ce sudo systemctl start docker; sudo systemctl enable docker sudo groupadd docker; sudo usermod -aG docker $USER â Logout and Login again and test docker docker version; docker run hello-world 20. Docker Versions - In 2017 the Docker versioning, release cycle, and product names changed - Docker Engine (the free one) is now Docker CE (Community Edition) - Docker Data Center is now Docker EE (Enterprise Edition) and adds additional paid products and support on top of Docker - Docker's version is now YY.MM based, using the month of its expected release, and the first one will be 17.03.0 - We now have two release tracks (called variants) "Edge" and "Stable". â Edge is released monthly and supported for a month. Quick and Easy Installation: https://get.docker.com/ â Stable is released quarterly and support for 4 months. 21. Docker Compose - Compose is a tool for defining and running multi- container Docker applications. With Compose, you use a YAML file to configure your applicationâs services. - Install Docker Compose on Linux â https://docs.docker.com/compose/install/#install-compose sudo curl -L "https://github.com/docker/compose/releases/download/1.23.1/docker- compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose sudo chmod +x /usr/local/bin/docker-compose sudo curl -L https://raw.githubusercontent.com/docker/compose/1.23.1/contrib/completion/bash/doc ker-compose -o /etc/bash_completion.d/docker-compose docker-compose --version 22. Docker Machine - Docker Machine is a tool that lets you install Docker Engine on virtual hosts, and manage the hosts with docker-machine commands. You can use Machine to create Docker hosts on your local Mac or Windows box, on your company network, in your data center, or on cloud providers like Azure, AWS, or Digital Ocean. - Install Docker Machine https://docs.docker.com/machine/install-machine/ base=https://github.com/docker/machine/releases/download/v0.16.0 && curl -L $base/docker-machine-$(uname -s)-$(uname -m) >/tmp/docker-machine && sudo install /tmp/docker-machine /usr/local/bin/docker-machine docker-machine version - Optional Bash Completion: https://docs.docker.com/machine/install- machine/#install-bash-completion-scripts 23. 4. Using Containers - Check Docker Server (Engine, Daemon) Running â docker version â docker info â docker-machine version â docker-compose version â docker run hello-world (First Hello World app) - How to get help on commands â docker (Lists all) â docker --help â https://docs.docker.com/ - New style 2017: Management Commands â docker run vs. docker container run â docker ps vs. docker container ls 24. Image vs. Container - An Image is the application we want to run - A Container is an instance of that image running as a process - You can have many containers running off the same image - How to get images? â Default image "registry" is called Docker Hub (hub.docker.com) - Containers arenât Mini-VMâs. They are just processes - Limited to what resources they can access (file paths, network devices, running processes) - Exit when process stops 25. Start a Simple Web Server - Let's start a simple web server nginx as container docker container run --publish 80:80 nginx â First look for the nginx image locally â if not found pull from Docker Hub â Start nginx and open port 80 on the host â Routes traffic to the container IP on port 80 â Start a firefox and check localhost, refresh couple of times â if you have bind error => Apache or another Web Server running. Stop & Disable with systemctl or choose another port on the host like --publish 8080:80 26. Nginx Lab - Now nginx running on the foreground and displaying logs on the terminal. Let's run it on the background â Hit Control-c docker container run --publish 80:80 --detach nginx â Now running at background, unique Container ID â Check running containers docker ps (Old way) docker container ls (New way) â Notice random funny names at the end, we can specify names too. Now stop nginx docker container stop or docker container ls (No running containers) docker container ls -a (Running and Stopped containers) 27. Nginx - Let's start a new container and give name "webhost" docker container run --publish 80:80 --detach --name webhost nginx docker container ls -a (Show both running and stopped) â Start firefox and refresh couple of times â Now let's check logs generated from container docker container logs webhost â Check running processes on "webhost" container docker container top webhost â Clean everything. All running and stopped containers docker container rm or (-f option to force) 28. Docker Internals - How Dockers implemented on Linux? â Docker uses several Linux kernel properties like namespaces, cgroups, and UnionFS 29. Docker Internals - Docker Engine uses the following namespaces on Linux: â PID namespace for process isolation. â NET namespace for managing network interfaces. â IPC namespace for managing access to IPC resources. â MNT namespace for managing filesystem mount points. â UTS namespace for isolating kernel and version - Docker Engine uses the following cgroups: â Memory cgroup for managing accounting, limits and notifications. â HugeTBL cgroup for accounting usage of huge pages by process group. â CPU group for managing user / system CPU time and usage. â CPUSet cgroup for binding a group to specific CPU. Useful for real time applications and NUMA systems with localized memory per CPU. â BlkIO cgroup for measuring & limiting amount of blckIO by group. â net_cls and net_prio cgroup for tagging the traffic control. â Devices cgroup for reading / writing access devices. â Freezer cgroup for freezing a group. Useful for cluster batch scheduling, process migration and debugging without affecting prtrace. 30. Containers are Processes - Let's start a simple container and check processes on the host. Redis is an open source key-value store that functions as a data structure server. docker container run --name mongo -d mongo docker container ls docker container top mongo (List of processes inside) - Now check processes on the host machine ps -ef | grep mongo (Also check parents with pstree -s -p and ps -efT for threads) - Stop mongo and check again docker container stop mongo ps -ef | grep mongo - Start mongo and check again, finally stop docker container start mongo 31. Exercise - Start and Inspect 3 Containers - Start an nginx, a mysql, and a httpd (apache) server â Use logs, inspect, and stats to check details docker container inspect docker container logs docker container stats (Like top utility) - Run all of them --detach (or -d), name them with --name - nginx should listen on 80:80, httpd on 8080:80, mysql on 3306:3306 - When running mysql, use the --env option (or -e) to pass in MYSQL_RANDOM_ROOT_PASSWORD=true - Use docker container logs on mysql to find the random password it created on startup - Use docker container ls to ensure everything is correct - LAB: Write a shell script to stop and remove all containers 32. Shell Access to Container - How to get a shell access to containers, using ssh? â Each container starts with a default command and stops when that command exists, you can change it â Also you can use -i -t to get an interactive shell docker container run -it -p 80:80 --name nginx nginx bash docker container top nginx â It does not make sense to start bash on nginx container but you can use exec to run additional command on any started container docker container run -d -p 80:80 --name ng nginx docker container exec -it ng bash docker container top ng (you will see nginx and bash processes) â LAB: Change index.html file and reload firefox to reflect changes 33. Shell Access Examples - Let's start a Ubuntu Container with an interactive shell â Note that default command for Ubuntu is already bash docker container run -it --name ubuntu ubuntu bash apt-get update dpkg -l | grep curl apt-cache search curl apt-get install curl curl www.google.com exit â Now how to start and re-connect to it? docker container ls -a docker container -ai start ubuntu - Also there is another mini-Linux distro called alpine ~5mb size! docker container run -it --name alpine alpine bash (No Bash!) docker container run -it --name alpine alpine sh 34. 5. Docker Networks - Network Types â bridge: The default network driver. If you donât specify a driver, this is the type of network you are creating â host: For standalone containers, remove network isolation between the container and the Docker host, and use the hostâs networking directly â none: For this container, disable all networking â overlay: Overlay networks connect multiple Docker daemons together and enable swarm services to communicate with each other. You can also use overlay networks to facilitate communication between a swarm service and a standalone container, or between two standalone containers on different Docker daemons 35. Bridge Network - Each container connected to a private virtual network "bridge" - Each virtual network routes through NAT firewall on host IP - All containers on a virtual network can talk to each other w/o -p - Best practice is to create a new virtual network for each app: 36. Best Practice - Default network is "bridge" for all created container - Inside of the bridge network container can see each other - However best practice is to create a new virtual network for the containers that needs to work together. There could be couple of seperated virtual networks. These networks are isolated and containers see each other over open ports - Using a seperate virtual network (not using the default bridge) has advantages. One of them is automatic DNS service. Each container within the same network can see each other with names instead of IP address, which is recommended way of operating since IP addresses can be changed frequently 37. Practice: Default Bridge Network - Practice of using default bridge network â Let's create an alpine container in detach mode docker container run -dit --name alpine1 alpine ash â alpine is a tiny linux distro, whenever we want to access it use: docker attach alpine1 and Control-pq to detach again â Now on another terminal attach and check IP docker attach alpine1 ifconfig (You see default network IP 172.17.0.2) â Create a second alpine and ping each other docker container run -dit --name alpine2 alpine ash docker attach alpine2 ifconfig ping 172.17.0.2 38. Practice: Create New Virtual Network - Check Network and Containers from Host Terminal docker network ls docker network inspect bridge â Look at the Container Section you will see alpine1,2 with IP addresses. Now create a new net: "alp-net" and alpine3 container on this net docker network create --driver bridge alp-net docker network ls docker container run -dit --name alpine3 --network alp-net alpine ash docker network inspect alp-net docker attach alpine3 ifconfig (You will see new ip 172.18.0.x) 39. Practice: DNS on Virtual Network - Now alpine1 and alpine2 on the default "bridge" and alpine3 on the alp-net. How to make alpine2 see alpine3? Create another network interface for alpine2 on the alp-net docker network connect --help docker network connect alp-net alpine2 docker network inspect alp-net (See Container Section) docker attach alpine2 ifconfig ping 172.18.0.2 (alpine2 can ping alpine3 IP using alp-net) ping alpine3 (DNS Service enables using hostnames) ping alpine1 (DNS is not available for default bridge) 40. LAB: Virtual Networks and DNS - Create a new virtual net: os-net - Create ubuntu:14.04 container with name ubuntu and -- rm option on os-net with -it and install curl - Create centos:7 container with name centos and --rm option on os-net with -it and install curl - Check that you can ping each other with hostname ubuntu and centos because of DNS Service - Create and nginx with name nginx on the default net with -d option - Try to ping nginx host with ip or hostname - Try to access with curl using ip connected to host - Exit both ubuntu and centos and check auto deletion 41. 6. Docker Images https://hub.docker.com/explore/ 42. What is an Image? - Official definition: "An Image is an ordered collection of root filesystem changes and the corresponding execution parameters for use within a container runtime." - Images are App binaries and dependencies - Not a complete OS. No kernel, kernel modules - Small as one file, Big as a CentOS Linux with yum, Nginx, Apache, MySQL, Mongo, etc. - Usually use a Dockerfile to create them - Stored in your Docker Engine image cache - Permanent Storage in Image Registry => hub.docker.com 43. Explore Docker Images - Create a free account at https://hub.docker.com/ and login, so you can create public repositories and only one private repository. You can choose paid plans to have more private repositories. - Hit Explore to view official images. Official images have approved by Docker Inc. with only names and "official" tag. When you create an image, it should have / - Choosing the right image: Search for Nginx and choose the image with "official" tag and lots of pulls and stars - Goto Details and Check Tags. To ensure the current version choose the "latest" tag which is default. 44. Practice: Pull Images - Let's pull different versions of Nginx â Goto Docker Hub and find official Nginx â Check images on the host (Remove if already exist) docker image ls docker pull nginx (Pulls the tag:latest) docker pull nginx:1.15.7 docker image ls â Notice the speed, not downloading everything â Check image size are identical but not consume disk â Also there are different tags like alpine docker pull nginx:alpine 45. Image Layers - Images are made up of file system changes and metadata - Each layer is uniquely identified and only stored once on a host using SHA and UnionFS (like zfs) - This saves storage space on host and transfer time on push/pull - A container is just a single read/write layer on top of image using COW - Use docker image history and inspect to see details docker image history nginx:latest docker image inspect nginx:latest 46. Image Layers 47. Image Layers 48. Docker Image Upload - How to tag and upload an image to Docker Hub? â Use nginx image first tag and upload docker image tag nginx trial/nginx (latest default) docker image ls (Notice exactly same as official) docker image push trial/nginx â Denied! You need to login with free docker account docker login => user/pass â WARNING! Your password will be stored unencrypted in /home/admin/.docker/config.json â Don't forget to docker logout to remove credentials docker image push trial/nginx docker image tag nginx trial/nginx:testing docker image push trial/nginx:testing (Same image fast) 49. 7. Dockerfile - Dockerfile is recipe for creating Docker Image - Dockerfile basics â FROM (base image) â ENV (environment variable) â RUN (any arbitrary shell command) â EXPOSE (open port from container to virtual network) â CMD (command to run when container starts) â docker image build (create image from Dockerfile) 50. Practice: Build Image from Dockerfile - Let's create an image using a sample Dockerfile cd dockerfile-sample-1 vim Dockerfile docker build -t mynginx . â Notice that we tagged as mynginx since we want to use locally, no need to specify username/repo â if you want to specify another Dockerfile use -f â Order is important, try to make minimal changes, let's edit Dockerfile and add port 8080 on EXPOSE docker build -t mynginx . â Very Fast Deployment since everything else is ready 51. Practice: Build Images and Push to Hub - Let's use the official nginx image and copy an index.html to create our own image, push it to Docker Hub cd dockerfile-sample-2 vim index.html (Change it as you like) vim Dockerfile (You can see only index.html copies) docker build -t mynginx:hello . docker run -p 80:80 --rm mynginx:hello docker image tag mynginx:hello trial/nginx:testing docker push trial/nginx:hello docker image rm trial/nginx:hello (Delete local images) docker run -p 80:80 --rm nginx (Hit control-c, auto rm) docker run -p 80:80 --rm trial/nginx:hello 52. 8. Data Volumes and Bind Mounts - Containers are usually immutable and ephemeral - "immutable infrastructure": only re-deploy containers, never change - This is the ideal scenario, but what about databases, or unique data? - Docker gives us features to ensure these "separation of concerns". This is known as "persistent data" - Two ways: Volumes and Bind Mounts â Volumes: Special location outside of container UFS â Bind Mounts: Link container path to host path 53. Volumes vs. Mounts - With Volume, a new directory is created within Docker's storage directory on the host machine, and Docker manages that directory's content. - Volumes are easier to back up or migrate than bind mounts. - You can manage volumes using Docker CLI commands or the Docker API. - Volumes work on both Linux and Windows containers. - Volumes can be more safely shared among multiple containers. - Volume drivers allow you to store volumes on remote hosts or cloud providers, to encrypt the contents of volumes, or to add other functionality. - A new volumeâs contents can be pre-populated by a container. 54. Bind Mounts - With Bind Mount, a file or directory on the host machine is mounted into a container. The file or directory is referenced by its full or relative path on the host machine. - Available since the early days of Docker. - Bind mounts have limited functionality compared to volumes. The file or directory does not need to exist on the Docker host already. It is created on demand if it does not yet exist. - Bind mounts are very performant, but they rely on the host machineâs filesystem having a specific directory structure available. - If you are developing new Docker applications, consider using named volumes instead. You canât use Docker CLI commands to directly manage bind mounts. 55. Practice: Volumes - Let's explore volume operations using mysql database - First check stop and remove all containers and delete existing volumes from previous work myrm => docker container rm -f docker volume list â if you ever run mysql there should be some anonymous volumes left, since deleting a container do not remove volumes. Default Location of volumes: /var/lib/docker/volumes. Let's delete all for a fresh start docker volume prune 56. Practice: Volumes - Goto Docker Hub and Check MySQL Dockerfile about volume info => VOLUME /var/lib/mysql - Create two a mysql container and check volume names docker pull mysql docker image inspect mysql (Check Volume) docker container run -d --name mysql1 -e MYSQL_ALLOW_EMPTY_PASSWORD=True mysql docker volume ls docker volume inspect (No info about container name) docker container run -d --name mysql2 -e MYSQL_ALLOW_EMPTY_PASSWORD=True mysql docker volume ls (We have a problem, use named volumes) 57. Practice: Volumes - Clean and create two new container with named volumes myrm; docker volume prune docker container run -d --name mysql1 -e MYSQL_ALLOW_EMPTY_PASSWORD=True -v mysql- db1:/var/lib/mysql mysql docker container run -d --name mysql2 -e MYSQL_ALLOW_EMPTY_PASSWORD=True -v mysql- db2:/var/lib/mysql mysql docker volume ls - Now stop and remove mysql2 and create mysql3 with mysql-db2, since volumes are not auto deleted with containers docker container rm -f mysql2 docker container run -d --name mysql3 -e MYSQL_ALLOW_EMPTY_PASSWORD=True -v mysql- db2:/var/lib/mysql mysql 58. Practice: Bind Mount - Clean all and create nginx1 and nginx2 containers â For nginx1 -p 80:80 manually connect to container and edit index.html â For nginx2 -p 8080:80 create a bind mount from host to container and change index.html see result docker container run -d --name nginx1 -p 80:80 nginx â Open browser on localhost and see test page docker container exec -it nginx1 bash # echo 'Welcome to Mars!' > /usr/share/nginx/html/index.html â Reload the browser 59. Practice: Bind Mount - Create bind mount from host to nginx2 to achieve same thing w/o login into container cd dockerfile-sample-2 docker container run -d --name nginx2 -p 8080:80 -v $(pwd):/usr/share/nginx/html nginx echo 'Welcome to Venus!' > index.html â Open browser http://localhost:8080 echo 'Welcome to Jupiter!' >> index.html â Reload browser. Very effective! â However Host specific and can not specify in Dockerfile 60. 9. Docker Compose - What is it? Why do we need it? â Standalone Container App is not a real world scenario â You need many Containers working together â How do we specify all details about configurations, volumes, networks, etc.? Obviously not with the command line docker options â Docker Compose comes into act right there - Docker compose consist of two parts: â YAML-formatted file that describes our solution options for: Containers, networks, volumes â A CLI tool docker-compose used for local dev/test automation with those YAML files - You need to install Docker Compose on Linux seperately â https://docs.docker.com/compose/install/#install-compose 61. YAML - YAML: YAML Ain't Markup Language => http://yaml.org - What It Is: YAML is a human friendly data serialization standard for all programming languages - There is a default name for Docker: docker-compose.yml â if you want use other names you need -f options with docker-compose command. Similar idea with Dockrfile and docker command - In terms of YAML versions definetely use v2 or higher â Details : https://docs.docker.com/compose/compose- file/compose-versioning/ - docker-compose.yml can be used with docker directly in production with Swarm (as of v1.13) 62. docker-compose CLI - CLI tool comes with Docker for Windows/Mac, but separate - download for Linux and not a production-grade tool but ideal for local development and test - Two most common commands are â docker-compose up # setup volumes/networks and start all containers â docker-compose down # stop all containers and remove cont/vol/net - Compose can also build your custom images â Will build them with docker-compose up if not found in cache â Also rebuild with docker-compose build or all in one: docker-compose up --build Great for complex builds 63. Template YAML File version: '3.1' # if no version is specificed then v1 is assumed. Recommend v2 minimum services: # containers same as docker run servicename: # a friendly name. this is also DNS name inside network image: # Optional if you use build: command: # Optional, replace the default CMD specified by the image environment: # Optional, same as -e in docker run volumes: # Optional, same as -v in docker run servicename2: volumes: # Optional, same as docker volume create networks: # Optional, same as docker network create 64. Sample YAML File version: '2' services: wordpress: image: wordpress ports: - 8080:80 environment: WORDPRESS_DB_HOST: mysql WORDPRESS_DB_NAME: wordpress volumes: - ./wordpress-data:/var/www/html mysql: image: mariadb environment: MYSQL_ROOT_PASSWORD: examplerootPW MYSQL_DATABASE: wordpress volumes: - mysql-data:/var/lib/mysql volumes: mysql-data: 65. Practice1: Create a docker-compose.yml - Goal: Create a compose config for a local Drupal CMS website - This empty directory is where you should create a docker- compose.yml - - Use the `drupal` image along with the `postgres` image - - Set the version to 2 - - Use `ports` to expose Drupal on 8080 - - Be sure to setup POSTGRES_PASSWORD on postgres image - - Walk though Drupal config in browser at http://localhost:8080 - - Tip: Drupal assumes DB is localhost, but it will actually be on the compose service name you give it - - Use Docker Hub documentation to figure out the right environment and volume settings 66. Practice1: docker-compose.yml version: '2' services: drupal: image: drupal ports: - "8080:80" volumes: - drupal-themes:/var/www/html/themes postgres: image: postgres environment: - POSTGRES_PASSWORD=mypasswd volumes: drupal-themes: 67. Practice2: Docker Compose Build - In YAML file you can specify build if you want to create your own images. Here is an example: Goto Practice2 folder: cat docker-compose.yml version: '2' services: proxy: build: context: . dockerfile: nginx.Dockerfile ports: - '80:80' web: image: httpd volumes: - ./html:/usr/local/apache2/htdocs/ cat nginx.Dockerfile FROM nginx:1.13 COPY nginx.conf /etc/nginx/conf.d/default.conf docker-compose up and docker-compose down --rmi local 68. 10. Swarm - How do we automate container lifecycle? - How can we easily scale up/down? - How can we ensure our containers are re-created if they fail? - How can we replace containers without downtime (blue/green deploy)? - How can we control/track where containers get started? - How can we create cross-node virtual networks? - How can we ensure only trusted servers run our containers? - How can we store secrets, keys, passwords and get them to the right container (and only that container)? 69. Swarm Mode: Built-In Orchestration - Swarm Mode is a clustering solution built inside Docker - Not related to Swarm "classic" for pre-1.12 versions - Added in 1.12 (Summer 2016) via SwarmKit toolkit - Enhanced in 1.13 (January 2017) via Stacks and Secrets - Not enabled by default, new commands once enabled â docker swarm, docker node, docker service â docker stack, docker secret - docker swarm init => Enabled! What Happened? â Lots of PKI and security automation, Root Signing Certificate created for our Swarm, Certificate is issued for first Manager node â Join tokens are created, Raft database created to store root CA, configs and secrets, Encrypted by default on disk (1.13+) â No need for another key/value system to hold orchestration/secrets, Replicates logs amongst Managers via mutual TLS in "control plane" 70. Manager and Worker Nodes 71. Nodes and Raft 72. Swarm Service 73. Docker Service Create 74. Docker Machine 75. Docker Machine and Swarm 76. Practice: Enable Swarm in Single Node - Check Swarm status and enable docker info | grep -i swarm (inactive) â Enable swarm docker swarm init (Error: Multiple interfaces, select one) docker swarm init --advertise-addr 192.168.56.111 â Success => Swarm initialized: current node (oz14e3meqzbwfdgtja3hh01sp) is now a manager. â To add a worker to this swarm, run the following command: docker swarm join --token SWMTKN-1- 2tlp9h62eqmendsqhm05f137w68jgwaeje66w2patt8gnd17b 0-0blsc43iaemb6w9u6871sxhes 192.168.56.111:2377 â To add a manager to this swarm, run 'docker swarm join- token manager' and follow the instructions. 77. Practice: Single Node Swarm - Check nodes docker node ls (One node, manager => Leader) - docker service create replaces docker container run in swarm mode â Create a service alpine, name it "homer", single replica docker service create --name homer alpine ping 8.8.8.8 docker service ls â Service "homer" is running with only 1 replica â Use docker service ps to get which node it is running docker service ps homer docker container ls docker container logs docker service logs 78. Practice: Single Node Swarm - Now, make it 3 replicas docker service update --replicas 3 homer docker service ls (Check all up: 3/3) docker service ps homer (Which is running on which) - Let's remove one container manually with docker container rm -f and see if swarm re-creates docker container ls docker container rm -f docker service ls (if you don't see, give a little time) - Remove service, see all 3 containers removed docker service rm homer docker service ls docker container ls 79. How to Create 3 Nodes Swarm? - A. play-with-docker.com â Only needs a browser, but resets after 4 hours - B. Local install with docker-machine + VirtualBox â Free and runs locally, but requires a machine with 8GB memory - C. Digital Ocean /AWS/Google Cloud + Docker install â Most like a production setup, but costs monthly - D. Create your own on the Cloud with docker-machine â docker-machine can provision machines for Amazon, Azure, Google - Finally, Install docker anywhere with get.docker.com 80. Practice: Multi Node Swarm - Goto https://labs.play-with-docker.com/ - Spin-up 3 machines: node1, node2, node3 - Login node1 and ping others ping docker info | grep -i swarm docker swarm init docker swarm init --advertise-addr 192.168.0.43 - On Node2 and join as worker (Later we will convert to Manager) docker swarm join --token SWMTKN-1- 0xb15jzxv2zvp45d9mrbvmnvnlf9zs9h2nxqone5tqjb5uvmte- 2q1e92xf2mvwdzjyk5keuicmc 192.168.0.43:2377 - On node1 run: docker node ls (node1 is manager:leader and node2 is worker) 81. Practice: Multi Node Swarm - On Node1: Promote node2 as Manager docker node update --role manager node2 docker node ls (Reachable) - Add Node3 as Manager directly â On Node1: docker swarm join-token manager â On Node3: docker swarm join --token SWMTKN-1- 0xb15jzxv2zvp45d9mrbvmnvnlf9zs9h2nxqone5tqjb5u vmte-2j66b6wafcym7p6uotgashohv 192.168.0.43:2377 â On Node1: docker node ls (Node3 also reachable) 82. Practice: Multi Node Swarm - Create a service again with alpine and 3 replicas docker service create --name homer --replicas 3 alpine ping 8.8.8.8 docker service ls docker node ps [node2] docker service ps homer (To see containers on nodes) - On node2 remove container, check recovery docker container ls docker container rm -f - On node1 check service and remove docker service ls docker service update --replicas 5 homer docker service ps homer docker service rm homer 83. 11. Swarm Network - Overlay Multi-Host Networking â Just choose --driver overlay when creating network â For container-to-container traffic inside a single Swarm â Optional IPSec encryption on network creation â Each service can be connected to multiple networks - Routing Mesh â Routes ingress (incoming) packets for a Service to proper Task â Spans all nodes in Swarm â Uses IPVS from Linux Kernel â Load balances Swarm Services across their Tasks 84. Overlay Network 85. Routing Mesh - This works Two ways: â Container-to-container in a Overlay network (uses VIP) â External traffic incoming to published ports (all nodes listen) - This is stateless load balancing - This LB is at Layer 3, not Layer 4 - Both limitation can be overcome with: â Nginx or HAProxy LB proxy, or: â Docker Enterprise Edition, which comes with built-in L4 web proxy 86. Practice: Overlay Network - Create an overlay network "mydrupal" and start 2 service: drupal and postgres. After you start check on all: curl http://localhost docker network create --driver overlay mydrupal docker network ls docker service create --name psql --network mydrupal -e POSTGRES_PASSWORD=mypass postgres docker service ls docker service ps psql docker container logs psql docker service create --name drupal --network mydrupal -p 80:80 drupal docker service ls docker service ps drupal docker service inspect drupal 87. Practice: Routing Mesh - Create a search app elasticsearch 3 replicas, each container has different initial string. Run curl http://localhost:9200 on different nodes. Observe it doesn't matter which node you run, always load balancing on existing nodes docker service create --name search --replicas 3 -p 9200:9200 elasticsearch:2 docker service ps search Node2> curl http://localhost:9200 Node3> curl http://localhost:9200 Node1> watch curl http://localhost:9200 88. Appendix A. Stack and Secret - In 1.13 Docker adds a new layer of abstraction to Swarm called Stacks - Stacks accept Compose files as their declarative definition for services, networks, and volumes - Use docker stack deploy rather then docker service create - Stacks manages all those objects for us, including overlay network per stack. - New deploy: key in Compose file. Can't do build: - Compose now ignores deploy:, Swarm ignores build: - docker-compose cli not needed on Swarm server 89. Docker Stack vs Docker Compose - Conceptually, both files serve the same purpose - deployment and configuration of your containers on docker engines. - Think docker-compose for developer tool on your local machine and docker stack as deployment tool on Swarm. - Docker-compose tool was created first and its purpose is "for defining and running multi-container Docker applications" on a single docker engine. - You use docker-compose up to create/update your containers, networks, volumes and so on. - Where Docker Stack is used in Docker Swarm (Docker's orchestration and scheduling tool) and, therefore, it has additional configuration parameters (i.e. replicas, deploy, roles) that are not needed on a single docker engine. - The stack file is interpreted by docker stack command. This command can be invoked from a docker swarm manager only - Specify a group of Docker containers to configure and deploy two ways: â Docker compose (docker-compose up) â Docker swarm (docker swarm init; docker stack deploy --compose-file docker-stack.yml mystack) 90. Stack 91. Secret Storage - Easiest "secure" solution for storing secrets in Swarm - What is a Secret? â Usernames and passwords â TLS certificates and keys â SSH keys â Supports generic strings or binary content up to 500kb - As of Docker 1.13.0 Swarm Raft DB is encrypted on disk - Only stored on disk on Manager nodes - Default is Managers and Workers "control plane" is TLS + Mutual Auth - Secrets are first stored in Swarm, then assigned to a Service(s) - Only containers in assigned Service(s) can see them 92. Practice: Voting App Stack Example - Let's create and run full swarm stack app designed as an example by Docker. You can check details: https://github.com/dockersamples/example-voting-app - First open https://labs.play-with-docker.com/ and create 5 node Managers using template just to avoid manual swarm setup we have done earlier - On Manager1 explore Swarm and Copy voting.yml file from your local machine to Manager1 with Drag and Drop cat voting.yml docker node ls docker service ls docker stack ls docker stack deploy -c voting.yml voteapp 93. Practice: Voting App Stack Example 94. Practice: Voting App Stack Example - On Manager1 explore voting app â First you see ports running 5000, 5001, 8080 â Open Chrome first on 5000 to vote â Check result on 5001. Open firefox and vote again â Finally look 8080 visualizer to see which service is running on which node docker stack ls docker stack ps voteapp docker stack services voteapp docker network ls â Now change voting.yml and change vote replicas to 5. Deploy again (it will update) Finally look at the visualizer docker stack deploy -c voting.yml voteapp 95. Practice: Secrets - Let's create secrets on the command line for postgres service and then do the same for stack in the yaml file. Do it on the swarm manager node. â Two ways to create: Use file or command line echo "mypsqluser" > psql_user.txt docker secret create psql_user psql_user.txt echo "mysecretpass123" | docker secret create psql_pass - docker secret ls docker secret inspect psql_user docker service create --name psql --secret psql_user --secret psql_pass -e POSTGRES_PASSWORD_FILE=/run/secrets/psql_pass -e POSTGRES_USER_FILE=/run/secrets/psql_user postgres docker service ps psql (Learn node and docker container ls) docker exec -it psql.1. bash cat /run/secrets/psql_user; cat /run/secrets/psql_pass; exit docker service rm psql 96. Practice: Secrets - Now let's copy docker-compose.yml psql_password.txt psql_user.txt to one of the manager node with drag and drop version: "3.1" services: psql: image: postgres secrets: - psql_user - psql_password environment: POSTGRES_PASSWORD_FILE: /run/secrets/psql_password POSTGRES_USER_FILE: /run/secrets/psql_user secrets: psql_user: file: ./psql_user.txt psql_password: file: ./psql_password.txt 97. Practice: Secrets - Now we can deploy our db service with stack using secret. Note that we need to use yaml version 3.1 for secrets. Also for stacks version should be at least 3. docker stack deploy -c docker-compose.yml mydb docker secret ls docker stack ls docker service ls docker service ps mydb_psql docker stack rm mydb https://www.friendbookmark.com/videos/900/hands-on-kubernetes-container-orchestrationThis Slide Presented in May 2019 at the "Cluster and Grid Computing" course at the "Iran University Of Science at Technology" by Amir Hossein Sorouri. https://www.friendbookmark.com/videos/899/why-i-love-kubernetes-failure-stories-and-you-should-too-goto-berlinTalk held on 2019-10-24 at GOTO Berlin: Everybody loves failure stories, but maybe for the wrong reasons: Schadenfreude and Internet comment threads are the dark side; continuous improvement through blameless postmortems, sharing incidents, and documenting learnings is what motivated me to compile the list of Kubernetes Failure Stories. Kubernetes gives us a infrastructure platform to talk in the same "language" and foster collaboration across organizations. In this talk, I will walk you through our horror stories of operating 100+ clusters and share the insights we gained from incidents, failures, user reports and general observations. I will highlight why Kubernetes makes sense despite its perceived complexity. Our failure stories will be sourced from recent and past incidents, so the talk will be up-to-date with our latest experiences. https://gotober.com/2019/sessions/1129/why-i-love-kubernetes-failure-stories-and-you-should-too