Hey everyone and welcome to Security Live
Security Live is a show focused on AWS and AWS security partners solving challenges for our customers you can find us on Twitch YouTube and LinkedIn
In every show we bring on an AWS partner and we discuss various topics related to security

My name is Margo and I am one of your co-hosts.
I'm a security and compliance specialist with AWS,
and Rob...
Hey good morning , good afternoon, possibly good evening my name's Rob Hale and I lead AWS's partner security business in EMEA
and today we have Aqua Security joining us we have Tsvi Koren who is the Field CTO from Aqua Security

Aqua Security is the pioneer in securing AI and containerized cloud native applications from development to production and they provide solutions at each of those milestones,
multiple milestones through the lifecycle

hey welcome! perhaps you can do a better job than I did there introducing yourself
Thanks for having me glad to be here so yeah I'm the field CTO of Aqua and as you said Aqua has started around nine years ago
when Docker containers started to emerge in the industry and our job is to secure them so we secure containers cloud native applications
in containers, in functions, in Kubernetes so if you hear any of these terms and you're wondering how to secure those probably give Aqua a call.

Nine years ago...that feels like a long time ago,... it's changing
Yeh, honestly I don't even remember what happened before 9 years ago, but you know I've been in security for about 25 years now
I started in the tail end of the last millennium the reason is that I just found a job in a security startup and I just got into security ever since
do you remember to say that like servers were like pets and you give them names so then what was it we used to say about containers? that containers were like cattle ?

if you're vegetarian it isn't a great analogy but ..it was like this I do remember this about 10 years ago this trending in conferences,
one of the things that's interesting about containers is that you really don't manage them individually right?
you manage them in groups, you manage them in deployments, you manage them in scale sets. so unlike the server that was probably in your basement,
and you gave it the name and you uploaded the software onto it and you managed over time, and patched it over time
containers come and go... and there are some really interesting methods on how to secure them and how to build them securely
so you feed in Walter not necessarily Walter but you feed and you look after a server but containers you must herd is that it? kind of herding containers?
is that the right terminology?
yeah basically yeah, there is a container orchestration, so containers are usually orchestrated right? you don't just start an individual container unless you're doing some testing
so you would start a whole deployment and then all the containers should behave the same and they should all have the same setting and they can compensate one another
they can grow they can scale with the application so the idea is that you are not going to have a single point of failure anywhere in the environment I did have a customer once I remember
and he used to have his chickens on his server in his office, this is actually a good use case he liked them to warm it and he actually had nests and chickens sitting on them anyway moving into containers what recommendations would Aqua Security have
for customers who are trying to prime their applications
to be defended properly
and of course we've like advanced now with AI and everything
but I'm sure they're kind of some
core best practice and recommendations you guys would recommend
so one thing that's really important to understand is
that containers caused a a kind of whole strata of IT practice
to basically disappear or become minimized
and that's ops.
so in the old world you would design your piece of software
say maybe it's Java application or .net application
and then you would ask ops to build your server
and they would build the operating system
they would put all the necessary components on that server
and then you would put the application on it
and that server would live maybe for a number of years
what happens with containers
is that the people that design the application
those that build the application and put the container images
that will eventually become live containers
have control over the entire stack top to bottom
so the selection of the operating system
the selection of the base components
what goes into the application
even the the base operating system version
are now all part of the application design
so when I talk to my customers
that's what I kind of tell them
the way I present that is that
you know, an individual developer really didn't have any knowledge
of what version of bash was running on the server
that their application would eventually run on
but now because there is the component selection
and the design of the application is now done
when you're building the image you do care
or at least you have an impact about what version of bash
what version of the base operating system
what other components of the base operating system are going into the application
and unfortunately development organizations got basically that job got
thrust upon them and they were not ready for that right
because they you know developers know how to write code
they really don't know how to design the operational status of an application
and that brings some of the security challenges about managing containers
and it also brings the development teams the pe the people that build and design applications
to make good selections so the first thing that I would say is that when you're building an an image for containers the selection of the base image is really crucial if your application does not need a lot of base operating system services you might want to use a lean image there are a lot of vendors you know Google Docker many images alpine is a good distribution for a lean image that don't carry a lot of the superfluous components that you would have for full operating system so if you just need to run a you know small node application or if your application component only needs to do one thing just make sure that you don't put a lot of other things in that operating system because these things can then just sit there and become a target for attackers this is all about reducing attack surface which we used to do in server hardening on a more one-to-one kind of basis
and often you'd have gold images that you'd load up
but when you're talking about containers
you're not talking about five or 10 or 20 servers
you're talking about hundreds possibly thousands of containers
so controlling your your attack surface and your vulnerability space
really seems like a an important thing to get right at the early stages
absolutely. and one thing to remember is that once a container is launched
unlike a server
you're really not supposed to touch it
it really is not supposed to have any interactive access
you're not supposed to log on to a running container
which means that a lot of the software
that you would have on it
that service the administrator
shouldn't really be there as well
things like user ad, things like the password application
the thing that allows you to change your password
those those really don't need to be in a container
because nobody should be using them
and the only people that might use them
are actually attackers that are going to use those
in order to further the cause inside of your environment
are there any tools that help you to view what your images are
so your templates for creating those containers
and then remove those vulnerabilities
or those services that are running
so there are basically four ways
to counter vulnerabilities in containers
the first and basic thing is just not to have any vulnerable components
and of course the more components you have just statistically you will have more chances
of having more vulnerable components
the other thing to do is to just keep your components up to date
and again that is very different from traditional servers
because in traditional servers
you would keep the application in place
and you would just patch the operating system around them
like you would do update
to your laptop you know from time to time
without changing the fundamental software in your laptop
but with containers you have to build a new image
so when you really don't, never patch a container
you would build a new image with updated components
and then you would replace the running container
with a new version of the entire container
that now has the patched components
and hopefully your application does not persist data in that container
because that is another thing that containers
are not supposed to do
and if all that works well you will now have instantiated
a whole slew of new containers
that's supposed to do the same thing with updated components
the other two ways to deal with vulnerabilities
is first of all to accept it
and we live in a world where there is no chance
to get zero vulnerabilities anywhere
so the whole idea is to manage vulnerabilities
manage the risk level that the organization or the application can tolerate
so accepting a vulnerability basically saying
that it's okay that it's there
we might have some compensating controls
maybe on the network side maybe on the runtime side,
and then the fourth
thing which is where Aqua really comes in to play
is we can compensate for vulnerabilities and other risk areas
by putting very strict runtime controls over
the things that a container can and cannot do
and that will make it so that an attacker will not be able
to further their cause inside of the container
because they will hit a wall
so reducing the number of containers that the application
just uses in general is going to not allow attackers
to live off the land as we call it
but then putting more and more restrictions
on what the containers to do
and really tailoring it to what the application needs
will allow us to run the application safely
even in a high-scale containerized environment
and I think what you're describing
we often use the term shift left
so you're kind of minting in or pushing in security practices
on the left hand side of the cycle
rather than once the containers have moved into production
how are you seeing that evolving now
in aqua security with the advent of AI?
are you seeing people using AI practices
in the left hand side of their pipelines?
is it impacting the number of vulnerabilities that people are seeing?
Yes so AI actually introduces
a whole new category of vulnerabilities.
when people think about vulnerabilities it's either
CVEs so common vulnerability exposures the system
that we have to classify bugs in software that have security implications
or it is a vulnerability in the software design
so things like SQL injections
and things that might be discovered by web application firewall
AI brings a whole new set of challenges
it is how the model is resilient to prompt poisoning
how resilient the model is to the feedback loop
between the model and the prompt of the users
there's also the selection of the SDK that you're going to use
if you're going to have the application
call out to an external AI model
so all of these are new components
that we are really only seeing in the last year or so
and those have their own set of vulnerabilities
that we need to manage
and that we need to be aware of
as we build our application
if it is an AI enabled application
that's really interesting in terms of where customers
need to spend their time
so if you had kind of three tips for for customers
who are building or have developers or application owners
who are really pushing for containers
what would those kind of three tips
be in terms of things that to do and things not to do
especially in a world where we have
lots of drive for AI and cloud native databases
so the first thing and as we said earlier
is just use the leanest image that you can
as your base image
do not put any components that really don't need to be there
in the file system of your application
this is the one, and the differences are really stark
you can have a bloated image or a full operating system
that can have maybe 500-600 vulnerabilities
that you need to manage and account for
and be screamed at to fix
but if you if you use a leaner image
if you let's say an Alpine image
or the the UBI the universal basic image from Red Hat
it really down boils down to the single digits
so the differences are really really stark
so the first thing I would say is
to use the leanest image that you can
the second thing is and that's something that
not a lot of people are paying attention to
is that images live in a registry
so the way that containers work you would manufacture an image
you would put it in the registry it basically waits there
until something like Kubernetes is going to make use of the image
and because we are dealing with generations of image
generations of images that build up the application
because each time we patch or make a change in the application
there's a new image generated
those registries really become bloated
and we've seen in our customer base
that customers have registries
where for two three five six seven years
nobody has ever deleted an image
that is also a problem because those get scanned all the time
and we get dinged on applications that really are not in use anymore
but also attackers really love to have Kubernetes go into registries
and pull old images and run them
because they know that they are more vulnerable
because they know that there's more bugs in them
and because Kubernetes will happily run any image that it has access to
getting images off the registry
after a year after six months don't do it like on a daily basis but
if you have images in your registry that are more than six months old
really ask yourself do they really need to be there
we see that so much with containers and serverless as well
if you've got legacy containers and legacy things
like Lambda functions for example
you know just letting those build up
and you then have to try and find out
if this actually still in use
how can we track that,
nobody wants to cause any downtime
but also you have vulnerabilities potentially increasing
and then my personal perspective is what they call lift and shift
which is you know people want to run things in container it's easier
you can really run them at scale on AWS
in things like the Kubernetes services or in lambda
and what they do is they take a complete server
that maybe have lived for a number of years
take the entire disk take the entire operating system
and then convert that to an image
and try to run that in a in a container
and this creates a myriad of problems
because a lot of times it actually forces you to run as a higher privileged user
because on a server, in order to make use of all the operating system services
you really need to be root so it kind of pushes you towards
giving containers more privileges that they need
but then it also plays into the fact that you will have
500,600, vulnerabilities
a lot of superfluous components
a lot of openings for an attacker to if they get the foothold
with a misconfiguration or a vulnerability
or maybe even something that came from the supply chain
we we've seen instances where components
have been poisoned on GitHub and been pulled in
they now have a complete toolbox that they can use to further a cause into the cloud
containers kind of self-document so they will tell you where your control plane is
and if you have all those operating system capabilities
then an attacker can really make use of those and attack your cloud
do you find when you're working with customers
as a company Aqua Security are you somewhat agnostic
if they're using like a managed service
like let's say the managed elastic kubernetes service versus let's say deploying natively on an EC2 instance is there from a security perspective
Do you recommend handing over some of the practices to the cloud service provider in that example or is it kind of are you seeing the same behavior in both cases same problem same ...
Yeh, interestingly, Aqua precedes Kubernetes
about maybe five six years old as an adopted platform for running containers
aqua has been around since basically Docker started containers in 2015
so we've been through what was called the the orchestration war
so when we started the company and thought of the methods
and what we need to do in order to protect containers the orchestration really wasn't a thing so Aqua in its core can protect a container wherever
it could be Docker or Containerd on a a EC2 instance it can be in in the Kubernetes services, it can be in Fargate it can be in in anywhere that containers are running
but over time we had to get more capabilities
that will help you manage the risk of those services
kubernetes itself is, first of all Kubernetes runs in containers
and then Kubernetes also has hundreds of configurations
that can go wrong
the cloud itself has maybe hundreds of thousands of configurations that could go wrong
so everything we do in Aqua is in the service of the workload eventually the container
but in order to defend a container you need to know where it's coming from you need to know where it is
you need to know what the parameters you need to know the environment that it operates in
and therefore we also have the ability to look into Kubernetes
look into the cloud and find the settings and the things that affect containers
so that we can account for them as we do the defense on those containers
that's interesting because from a Kubernetes perspective or a Docker perspective
a lot of those containers are on a server maybe on prem
and they get migrated into the cloud so is there a kind of a recommended way
of I guess streamlining that lift and shift
so you have security mechanisms in place to reduce the the attack surface
and the vulnerabilities and and make sure you're you're looking at permissions
as they migrate to the cloud so kind of a model to stop you
just lifting and shifting to get the job done so you know if lift and shift
is taking a complete server right a monolithic old style server
and turning it into a container image once you've turned that into a container image
and I would urge everybody to kind of look at the application rearchitect it
so you don't you know pull it completely into an image
but once you have a container a container image
the nice thing about it is it it contains all its dependencies
so it can run anywhere so you can actually safely take a container image
and run it you know on Docker server on your desk and then migrate it into EKS
and then migrate it into Fargate
and it's going to operate in the exact same way
however in each one of those environments the container might have different properties
and different access that it is going to inherit from the orchestration
so for instance when once you when you run a container in Kubernetes
and if people have access to containers
just do a dump of the environment variables
and you will see you know the IP address of the Kubernetes API server
basically the management plane you will see the cloud labels you will sometimes see cloud information
so those get inherited by the orchestration system
and when we look at how to defend containers
we also need to defend against attackers using that metadata
that is just readily available to be read in a container
in order to affect the orchestration system around it
so there's nothing wrong in taking a well-built container image and moving into the cloud
but just be aware that as you do that the container might inherit more information
and we just got to be aware that that information is there and it's really not protected
as long as somebody has access to that container
great tips there, i loved your phrase there "orchestration wars"
I actually hadn't heard that before and I didn't realize Aqua Security was around longer than Kubernetes
which is actually pretty cool do you see, do you think people introduce unnecessary complexity?
I sometimes amusingly feel when I'm talking to customers that like they describe an application
and rather than describing the business capability they go and we want to use Kubernetes,
is there unnecessary complexity that sometimes is brought in when they're trying to rearchitect and how can that be avoided?
yeah so you know not t every application should run in containers we're seeing in our customer base we're seeing kind of hybrid applications
where there could be a kind of high-capacity short transaction that might run in Lambda it might transact versus a set of deployments that run in EKS
that doesn't need to persist data but need to persist transactions but then those will talk to maybe applications that run on EC2 which then persist data into either cloud stories or locally so my advice would be just instead of determining in advance
how you're going to run the application look at what the application needs if this is a high-capacity short transaction application maybe function is what you want to do if this is a application that need to persist data let's say you run an FTP server for old-timers or something that needs to store files a container is not a good use case for that because containers are, Kubernetes, once it moves a container from one place to the other it might lose the storage of the container unless you're using an external storage so when you architect an application think of yourself what is the flow of the data what needs to be persisted what is the length of the transaction how long do we need to sustain a transaction and then you'll kind of get an idea of what is the right deployment mode for that component of the application and applications you know to complete applications end-to-end applications could have certain modules that run in EC2 instances they might have modules that run in EKS and they might have modules that that run in Lambda Just a quick reminder if anyone's has questions on container security pop them in the in the chat and can ask them live and and discuss them and there's also been a QR code flashing up here we go which I believe Tsvi is linked to an Aqua security tutorial ?
yeah it's linked  into our information page about running generative AI safely in containers
but on the Aqua website once you get there we have our container wiki which is agood source of information for containers and best practices you have access to our knowledge base around how to safely run containers and there are links there to various resources if you want to learn more about how to run containers safely or the capabilities of a what do you think containers will look like five years from now we've talked about the past we've talked about the orchestration wars which sounds like an ep George or or Martin book by the way we've talked about aqua security being around longer than kubernetes which I love what do you think the next five years might look like so containers are now the primary vehicle to run AI enabled applications and AI models as and as organizations start to look into implementing their own onrem which doesn't doesn't mean in their building but but under their control AI models for their own uses the use of containers is is only going to grow um now you know I've been long enough you know even from the mainframe days and and you know back in the 90s people were telling us that mainframes are going to disappear uh that hasn't happened right they have very very long tail i think containers are going to be there in five years they are probably going to be orchestrated by uh AI capable orchestration which is going to be even more intelligent at what Kubernetes can can provide hopefully we will have a much better handle on the vulnerabilities in in containers but containers are not going away they are they are now the primary vehicle to build applications and I expect that to to stay the course for at least a number of years i love your use there of the phrase on premise it's the first time I've heard it used
but you used it in a context of somebody actually you know running their own model or doing their own training uh in a container maybe in the cloud but it's like this on premise notion we're doing it ourselves is that what you meant did I pick you up correctly?
yeah on-prem can mean different things but it really means under your control right so could be an you can your application can refer to chatgpt and or or you know lama as a as a as an AI model you can run a local copy of that you can run a local copy of that in your own AWS account or you can own the copy in your own data center but the container is going to be the same container it needs to be defended the same way yeah we have a we have a you know a similar model in terms of we bring the model to the customer in AWS services like bedrock for example so we bring that into your environment and then allow that to access data etc rather than sending data to the model that seems a you know a poor security and governance model of f operation yeah absolutely well I think this has been super interesting will you be at Rainforce which is happening uh next week yes next week I believe any chance we get you over there
i'm not sure that Aqua is  going to have a presence in Reinforce but we are always happy to engage with people that run services in a AWS and you can always contact Aqua if you have a need in that respect.
Awesome it's been awesome to have you thank you so much for your time really really great to hear about the things happening in containers and security and how our customers can can start to implement better security and more governance around that so thank you very much