Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: Cannot log into run container when ~/.ssh is located on a volume #2257

Closed
un-def opened this issue Feb 3, 2025 · 0 comments · Fixed by #2261
Closed

[Bug]: Cannot log into run container when ~/.ssh is located on a volume #2257

un-def opened this issue Feb 3, 2025 · 0 comments · Fixed by #2261
Assignees
@un-def
Labels
bug Something isn't working

Comments

@un-def
Copy link
Collaborator

un-def commented Feb 3, 2025

Steps to reproduce

  1. Prepare a run configuration with lots of env variables (either in env property on in a image itself) and /home/username located on a reusable volume
  2. Apply run configuration several times, so that N vars × M applications > 1000
  3. ssh username@runname

Actual behaviour

Cannot log in: MOTD is displayed, but the session is terminated immediately

Expected behaviour

No response

dstack version

0.18.39

Server logs



Additional information


There is a hardcoded upper limit in OpenSSH on the number of lines in ~/.ssh/environment, and dstack-runner appends new lines on each run.


https://github.com/openssh/openssh-portable/blob/53a80baaebda180f46e6e8571f3ff800e1f5c496/session.c#L846-L847

      		
    

      
  





        



            

              
            

        

      


      
    








      

    



      

  
            

    

      
    

    


      

          @un-def
un-def




          added
  the 

  bug

  Something isn't working
 label


      Feb 3, 2025

    

  



    

    

      
    

    


      

          @un-def
un-def




              self-assigned this


      Feb 3, 2025

    

  



  


  
    
  
  

    
  un-def 

      added a commit
      that referenced
      this issue

    
      Feb 4, 2025
    
    

      


  

    

      


  

      
        @un-def
  





      

        
          [runner] Back up and restore ~/.ssh files
        

          
            
          

      


      

          

    
    
    

  

      


      

          

    
    
    

  

  

    
  



      


      
      

        
          6ec169d
        
      

    

  

    

      
Preserve `~/.ssh` directory state, namely `environment` and
`authorized_keys` files, as follows:

Before a job started:

* If `<file>` exists:
    - rename `<file>` to `<file>.dstack.bak`
    - copy backup to a new file with original name `<file>`
* If `<file>` does not exist: remember this

After the job finished:

* If `<file>` existed: rename `<file>.dstack.bak` back to `<file>`
* If `<file>` did not exist: remove `<file>`

Fixes: #2257

    




    

  




  

  

    
  


  

      

    @un-def
un-def



    mentioned this issue
    
      Feb 4, 2025
    





  


  

      
   Merged

  







  




    

    


  


  
    
  
  

    
  un-def 

      added a commit
      that referenced
      this issue

    
      Feb 4, 2025
    
    

      


  

    

      


  

      
        @un-def
  





      

        
          [runner] Back up and restore ~/.ssh files (#2261)
        

          
            
          

      


      

          

    
    
    

  

    


      


      

          

    
    
    

  

  

    
  



      


      
      

        
          4cc82c3
        
      

    

  

    

      
Preserve `~/.ssh` directory state, namely `environment` and
`authorized_keys` files, as follows:

Before a job started:

* If `<file>` exists:
    - rename `<file>` to `<file>.dstack.bak`
    - copy backup to a new file with original name `<file>`
* If `<file>` does not exist: remember this

After the job finished:

* If `<file>` existed: rename `<file>.dstack.bak` back to `<file>`
* If `<file>` did not exist: remove `<file>`

Fixes: #2257

    




    

  




  


  
    
  
  

    
  pranitnaik43 

      pushed a commit
        to bahaal-tech/dstack
      that referenced
      this issue

    
      Feb 9, 2025
    
    

      


  

    

      


  

      
        @un-def
  





      

        
          [runner] Back up and restore ~/.ssh files (dstackai#2261)
        

          
            
          

      


      

          

    
    
    

  

      


      

      


      
      

        
          0fa471c
        
      

    

  

    

      
Preserve `~/.ssh` directory state, namely `environment` and
`authorized_keys` files, as follows:

Before a job started:

* If `<file>` exists:
    - rename `<file>` to `<file>.dstack.bak`
    - copy backup to a new file with original name `<file>`
* If `<file>` does not exist: remember this

After the job finished:

* If `<file>` existed: rename `<file>.dstack.bak` back to `<file>`
* If `<file>` did not exist: remove `<file>`

Fixes: dstackai#2257

    




    

  













  
  

    

      

  





  




  
  

              

  
    Sign up for free
    to join this conversation on GitHub.
    Already have an account?
    Sign in to comment


  



  






  
                  




    

  


      
  

    Assignees
  



        

      

        

  
      @un-def
  
    un-def
    

      








      


  


  

    Labels
  



    

      

    bug

  Something isn't working








      

  

    

        

    Projects
  


        




    None yet




  



      


  

    
  

    Milestone
  


      No milestone





    
      
          


  

    
      

        
    

    Development
  




          
  
Successfully merging a pull request may close this issue.





    


  
          
      [runner] Back up and restore ~/.ssh files

    
      dstackai/dstack
    





    
  




      

    

  
      

  

    

      1 participant
    

    

        
          @un-def